Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks

Hyunjae Kim; Hyeon Hwang; Jiwoo Lee; Sihyeon Park; Dain Kim; Taewhoo Lee; Chanwoong Yoon; Jiwoong Sohn; Donghee Choi; Jaewoo Kang

Source-linked AI summary

Small Language Models Learn Enhanced Reasoning Skills from Medical Textbooks

Hyunjae Kim, Hyeon Hwang, Jiwoo Lee, Sihyeon Park, Dain Kim, Taewhoo Lee, Chanwoong Yoon, Jiwoong Sohn, Donghee Choi, Jaewoo Kang

arXiv:2404.00376v2Mar 30, 2024cs.CL

TL;DR

Closed-source medical LMs raise privacy and security concerns, while smaller open-source models have lacked strong multi-step reasoning. Meerkat addresses this gap with textbook-derived synthetic chain-of-thought training and achieves broad gains, including a 1.3% advantage over GPT-4 at 70B scale, while factuality and deployment safety remain limitations.

Problem
Closed-source medical LMs create privacy and security concerns, while smaller open-source models lack the multi-step reasoning needed for complex medical problems.
Method
Meerkat fine-tunes 7B-to-70B open-source medical models with chain-of-thought data from 18 textbooks, MedQA reasoning paths, and instruction-following datasets.
Results
Across six benchmarks, Meerkat models outperformed prior small models and GPT-3.5, while Meerkat-70B exceeded GPT-4 by 1.3% and transferred competitively to clinical cases.
Takeaways & Limitations
Textbook-derived chain-of-thought training produced open-source medical models that substantially narrowed the performance gap with larger commercial LMs.
Takeaways & Limitations
Meerkat models had lower factuality than GPT-3.5 and GPT-4, lacked RLHF, and require expert validation because responses may be unsupported, unsafe, or biased.

Abstract

from arXiv · show

While recent advancements in commercial large language models (LM) have shown promising results in medical tasks, their closed-source nature poses significant privacy and security concerns, hindering their widespread use in the medical field. Despite efforts to create open-source models, their limited parameters often result in insufficient multi-step reasoning capabilities required for solving complex medical problems. To address this, we introduce Meerkat, a new family of medical AI systems ranging from 7 to 70 billion parameters. The models were trained using our new synthetic dataset consisting of high-quality chain-of-thought reasoning paths sourced from 18 medical textbooks, along with diverse instruction-following datasets. Our systems achieved remarkable accuracy across six medical benchmarks, surpassing the previous best models such as MediTron and BioMistral, and GPT-3.5 by a large margin. Notably, Meerkat-7B surpassed the passing threshold of the United States Medical Licensing Examination (USMLE) for the first time for a 7B-parameter model, while Meerkat-70B outperformed GPT-4 by an average of 1.3%. Additionally, Meerkat-70B correctly diagnosed 21 out of 38 complex clinical cases, outperforming humans' 13.8 and closely matching GPT-4's 21.8. Our systems offered more detailed free-form responses to clinical queries compared to existing small models, approaching the performance level of large commercial models. This significantly narrows the performance gap with large LMs, showcasing its effectiveness in addressing complex medical challenges.

1 Introduction

Meerkat addresses privacy concerns surrounding closed-source medical LMs and the limited multi-step reasoning of smaller open-source models. It introduces open-source systems trained with textbook-derived chain-of-thought data and reports strong benchmark performance, including new results for 7B and 70B models.

Meerkat targets privacy and security concerns from requiring sensitive medical data to pass through closed-source commercial APIs.
Smaller open-source medical LMs lack the multi-step reasoning needed for complex problems, motivating a model that acquires chain-of-thought abilities.
Meerkat is an open-source family built on Mistral-7B, LLaMA-3-8B, and LLaMA-3-70B and fine-tuned with diverse crafted data.
78K synthetic chain-of-thought examples were generated from 18 medical textbooks alongside 9.3K MedQA reasoning paths and instruction-following data.
77.1% on MedQA made Meerkat-7B the first 7B model to exceed the 60% USMLE passing threshold.
Meerkat-70B exceeded GPT-4 by 1.3% across six medical benchmarks, while Meerkat-7B and newer 8B and 70B systems advanced open-source performance.

2.1 Main Results

Meerkat models outperformed competing small medical models across six multiple-choice benchmarks and exceeded GPT-4 at 70B scale. They also transferred to challenging clinical cases and produced comprehensive real-world answers, although factuality remained below larger commercial models.

2.1.1 Multiple-Choice QA: Meerkat-70B surpassed GPT-4 by 1.3% across six medical benchmarks, establishing a new state-of-the-art result.
2.1.1 Multiple-Choice QA: Meerkat-7B and Meerkat-8B exceeded GPT-3.5 by 7.8% and 12.5%, respectively, while outperforming prior 7B baselines by 11.6% and 16.3%.
2.1.1 Multiple-Choice QA: 77.1% was Meerkat-7B’s ensemble MedQA score, exceeding the 60% USMLE threshold; Meerkat-70B reached 85.8% and exceeded GPT-4 by 4.4% in ensemble evaluation.
2.1.2 NEJM Case Challenges: On NEJM case challenges, Meerkat models outperformed prior instruction-tuned models and humans, while Meerkat-70B performed comparably to GPT-4.
2.1.3 Real-world Clinical Questions: Meerkat responses on K-QA achieved completeness comparable to or better than GPT-3.5 and all other open-source LMs, but factuality remained below GPT-3.5 and GPT-4.

2.2 Ablation Study

CoT fine-tuning and textbook-derived reasoning data progressively improved MedQA performance across six evaluated models.

2.2.1 Effect of CoT Fine-tuning: 7.5% average accuracy gain followed CoT fine-tuning across six models, compared with training on MedQA question-answer pairs alone.
2.2.2 Effect of Textbook Augmentation: 5.1% further average accuracy improvement followed adding textbook-generated QA pairs and CoT paths to MedQA-CoT training.
2.2.2 Effect of Textbook Augmentation: Figure 3 compares MedQA, MedQA-CoT, and MedQA-CoT + MedBooks-CoT-18 training across six language models.
2.2 Ablation Study: General-purpose Mistral-7B, Gemma-7B, and LLaMA-3-8B outperformed biomedical-specialized MediTron-7B and BioMistral-7B on MedQA.

2.3 Assessment of Model Explanations

Meerkat-7B generated explanations that most closely matched human explanations on lexical and semantic metrics, although GPT-4 received the highest overall evaluator score.

2.3 Assessment of Model Explanations: Meerkat-7B performed best on ROUGE-L [36] and BERTScore [37], indicating close lexical and semantic similarity to human explanations.
2.3 Assessment of Model Explanations: GPT-4 achieved the highest GPT-4 score, while Meerkat-7B ranked ahead of Mistral-7B-Instruct.
2.3 Assessment of Model Explanations: Explanations for correctly answered questions scored over 1.4 points higher than explanations for incorrect answers.

3 Discussion

Meerkat models are openly available small medical LMs trained with textbook-derived CoT data and generally outperform baseline models across medical evaluations.

3 Discussion: Fine-tuning with high-quality CoT data enabled Meerkat models to largely outperform baseline models across six benchmarks and NEJM case challenges.
3 Discussion: Meerkat models provide more detailed responses without sacrificing factuality relative to baseline small models, but their factuality decreases versus GPT-3.5 and GPT-4.
3 Discussion: The instruction-tuning dataset contains 460,625 training examples.
3 Discussion: The models were not fine-tuned with RLHF, leaving possible unsupported, unsafe, or biased responses and motivating expert validation before real-world deployment.

4 Methods

Meerkat models are open-source medical systems built from state-of-the-art open-source language models and instruction-tuned for medical applications.

4 Methods: Meerkat is based on state-of-the-art open-source models such as Mistral-7B and LLaMA-3 and is specifically instruction-tuned for medicine.
4 Methods: Its training uses a high-quality instruction-tuning dataset designed for diverse medical-domain applications.

4.1 Training

Meerkat training combines MedQA chain-of-thought data, textbook-derived synthetic reasoning examples, and instruction-following datasets. GPT-4 generates textbook-grounded questions and explanations to broaden coverage while reducing hallucination risk.

4.1.1 Chain-of-thought Reasoning Data Generation: GPT-4 first produced 9.3K step-by-step solutions for MedQA questions, then generated textbook-conditioned questions and explanations using sampled MedQA questions as references.The process yields an additional 78K chain-of-thought examples after generating synthetic question-answer pairs from textbook chunks.
4.1.2 Data Augmentation With Medical Textbooks: 78K synthetic question-answer pairs with chain-of-thought paths were generated from 18 medical textbooks to expand beyond MedQA’s training examples.The textbooks span disciplines including anatomy, internal medicine, neurology, pathology, pediatrics, pharmacology, psychiatry, and surgery.
4.1.2 Data Augmentation With Medical Textbooks: Textbook chunks were paired with three sampled MedQA questions, enabling GPT-4 to generate case-based questions, plausible options, correct answers, and reasoning paths grounded in source content.The authors state that textbook-based generation reduces potential hallucinations and improves coverage of essential textbook content.
4.1.3 Utilization of Existing Instruction-following Datasets: Training also incorporated instruction-following datasets covering broad medical knowledge, cleaned patient inquiries, and multi-turn medical dialogues.MedMCQA covers 21 medical subjects, while ChatDoctor-cleaned removes noise from 112K original responses using GPT-3.5.
4.1.4 Model Training: The models were initialized from Mistral-7B, LLaMA-3-8B-Instruct, and LLaMA-3-70B-Instruct weights and fine-tuned on nine combined training datasets with next-token prediction.The 7B model was trained for three epochs using eight 80G A100 GPUs.

4.2 Experimental Settings

The experiments evaluate Meerkat on multiple-choice medical benchmarks and free-form clinical responses using completeness and factuality measures. The setup also standardizes baseline training and inference procedures for comparison.

4.2.1 Benchmark Datasets: Six medical benchmarks use multiple-choice questions, while K-QA evaluates free-form clinical answers against human-annotated statements.Table 3 summarizes benchmark sizes and evaluation metrics, distinguishing K-QA from the multiple-choice datasets.
4.2.2 Completeness and Factuality Evaluation: K-QA completeness measures entailment of must-have statements, whereas factuality checks contradictions against must-have and nice-to-have statements.Both scores are averaged across all questions.
4.2.3 Baseline Training: Baseline models used different learning rates and three training epochs, with better-performing configurations selected because performance varied substantially with these choices.LLaMA-7B and MediTron-7B used 2e-5, while Mistral-7B, BioMistral-7B, and Gemma-7B used 2e-6.
4.2.4 Inference Settings: Single-model QA evaluation used BFloat16 with greedy decoding, while ensemble evaluation used temperature 0.7 and majority voting after shuffling answer choices.The choice-shuffling ensemble is intended to mitigate positional biases.

Appendix A Extended Data

The appendix provides extended benchmark statistics, prompt examples, and qualitative response comparisons. These materials document the evaluation datasets, synthetic-data generation procedures, and differences among human, GPT-4, and Meerkat responses.

Extended benchmark data: Table A1 reports model performance across six QA benchmarks and six medical subjects in MMLU-Medical.MMLU-Medical scores average accuracy across six medical-related subjects from the original MMLU benchmark.
Qualitative response comparisons: Tables A2–A4 compare human, GPT-4, ChatDoctor-7B, and Meerkat-7B responses on medication, empiric-treatment, and dosage-titration questions.The examples include both correct reasoning and a Meerkat dosage-duration error in the Saxenda case.
Synthetic-data prompts: Tables A5 and A6 show the prompts used to generate MedQA chain-of-thought data and textbook-derived synthetic QA data.Both procedures use GPT-4 checkpoints for generating answers, questions, and reasoning paths.
Instruction-following data prompts: Tables A7 and A8 document GPT-3.5 prompts for cleaning ChatDoctor responses and generating MedQA-style medical dialogues.The listed checkpoint is gpt-3.5-turbo-0613.