Source-linked AI summary
Towards Building Multilingual Language Model for Medicine
Pengcheng Qiu, Chaoyi Wu, Xiaoman Zhang, Weixiong Lin, Haicheng Wang, Ya Zhang, Yanfeng Wang, Weidi Xie
TL;DR
Non-English medical queries remain challenging for open-source multilingual LLMs because general multilingual corpora under-represent medical content. The paper develops MMedC and MMedBench, evaluates existing and further-trained models, and reports that MMed-Llama 3 performs best among open-source models across multilingual and English-only benchmarks.
Problem
Open-source multilingual LLMs exhibit unsatisfactory performance on non-English medical queries because general multilingual datasets under-represent medical content.
Method
The paper constructs MMedC, builds the rationale-enhanced MMedBench, and evaluates existing LLMs alongside models further trained autoregressively on MMedC.
Results
MMed-Llama 3 demonstrates the best performance on both multilingual and English-only benchmarks among the evaluated open-source models.
Takeaways & Limitations
Further auto-regressive training on MMedC improves model performance, supporting specialized multilingual medical corpora for adapting general LLMs to medical contexts.
Takeaways & Limitations
The corpus may contain biases against underprivileged populations because a significant portion of its data was acquired through web crawling.
Abstract
from arXiv · showhide
The development of open-source, multilingual medical language models can benefit a wide, linguistically diverse audience from different regions. To promote this domain, we present contributions from the following: First, we construct a multilingual medical corpus, containing approximately 25.5B tokens encompassing 6 main languages, termed as MMedC, enabling auto-regressive domain adaptation for general LLMs; Second, to monitor the development of multilingual medical LLMs, we propose a multilingual medical multi-choice question-answering benchmark with rationale, termed as MMedBench; Third, we have assessed a number of open-source large language models (LLMs) on our benchmark, along with those further auto-regressive trained on MMedC. Our final model, MMed-Llama 3, with only 8B parameters, achieves superior performance compared to all other open-source models on both MMedBench and English benchmarks, even rivaling GPT-4. In conclusion, in this work, we present a large-scale corpus, a benchmark and a series of models to support the development of multilingual medical LLMs.
1 Introduction
The paper addresses weak multilingual medical performance by introducing a large medical corpus, a rationale-enhanced benchmark, and evaluations of existing and further-trained language models.
- Multilingual open-source LLMs remain unsatisfactory on non-English medical queries, largely because general multilingual datasets under-represent medical content.
- MMedC contains over 25.5B medical tokens across English, Chinese, Japanese, French, Russian, and Spanish, compiled from four data sources.The sources include filtered multilingual text, medical textbooks, open-source medical websites, and existing medical corpora.
- MMedBench aggregates multilingual medical multiple-choice QA and GPT-4-generated rationales across 53,566 pairs spanning 21 medical fields.It includes 45,048 training pairs and 8,518 test pairs, with a manually verified rationale subset for reasoning evaluation.
- MMedBench evaluates multiple-choice accuracy and rationale generation under zero-shot, parameter-efficient fine-tuning, and full fine-tuning settings.
- The study benchmarks eleven existing multilingual-supporting LLMs alongside models further trained with MMedC.The evaluated systems include both closed- and open-source models across several medical and general-purpose model families.
- Further auto-regressive training on MMedC improves model performance, and MMed-Llama 3 achieves the best results among evaluated open-source models on multilingual and English-only benchmarks.
2 Results
The paper presents statistics for MMedC and MMedBench, then evaluates multilingual medical LLMs across question answering, rationale generation, and English benchmarks. MMed-Llama 3 achieves the strongest reported results, while ablations show benefits from comprehensive multilingual training data.
- Dataset statistics: MMedC contains over 25.5B tokens across six languages, with English comprising 42% and Russian 7% of the corpus.The corpus combines filtered multilingual content, medical textbooks, medical websites, and existing small-scale corpora.
- Benchmark statistics: MMedBench spans 21 medical fields and includes multi-option questions with rationale sections averaging 200 tokens.Its topic coverage ranges from foundational clinical medicine to specialized areas such as pharmacology and public health.
- Model evaluation: 67.75 versus 62.79 was the full-fine-tuning average accuracy for MMed-Llama 3 and Llama 3, respectively, on multilingual MMedBench.The reported comparison indicates a gain after additional auto-regressive training on MMedC.
- Model evaluation: MMed-Llama 3 achieved the highest human rating of 4.10 and GPT-4 rating of 4.73, exceeding InternLM 2 by 0.89 GPT-4-rating points.These ratings aligned with its superior automatic-metric performance.
- English benchmarks: MMed-Llama 3 obtained 4.5%, 4.3%, and 2.2% gains on MedQA, MedMCQA, and PubMedQA, respectively, and scored 72.59 versus GPT-3.5’s 67.69 on MMLU-Medical.The model was evaluated on four English medical multiple-choice benchmarks.
- Ablation studies: Adding comprehensive rationales increased average multiple-choice accuracy from 58.72 to 62.79, while full multilingual training raised it to 64.40 and US-Data further raised it to 67.75.English-only auto-regressive training did not improve overall accuracy and was conjectured to overfit English.
3 Discussion
The discussion finds that specialized multilingual medical data, rationale supervision, and stronger foundation models improve multilingual medical LLM development. It also identifies research and clinical opportunities while noting dataset bias, incomplete language coverage, limited explainability, and the computational constraints of the 8B model.
- Empirical conclusions: Auto-regressive training on MMedC improves MMedLM, MMedLM 2, and MMed-Llama 3 over their original baselines.The authors attribute these gains to targeted domain-specific knowledge and argue that existing pre-training corpora are limited for multilingual medical contexts.
- Empirical conclusions: 2.33%, 2.42% and 4.07%: rationale-enhanced fine-tuning improves MMedBench accuracy for InternLM, InternLM 2 and Llama 3, respectively.Joint training on multiple-choice prediction and rationale generation also enables rationale output.
- Empirical conclusions: Stronger foundation models generally improve multilingual medical QA, and their multilingual abilities remain useful after medical-domain adaptation.The discussion compares Llama 2, Mistral, and Llama 3, and extends the pattern to MMedLM, MMedLM 2, and MMed-Llama 3.
- Research and clinical impacts: Multilingual medical LLMs may support multilingual retrieval-augmented generation, communication across healthcare language barriers, and medical education.The proposed applications include broader multilingual medical knowledge retrieval, translation and interpretation, and educational materials or simulations in multiple languages.
- Limitations: The work is limited by web-crawled bias, incomplete global language coverage, under-explored explainability, and the computational cost of scaling beyond 8B parameters.The authors propose stronger bias controls, additional languages, larger retrieval-augmented models, and further explainability work.
4 Methodology
The methodology constructs MMedC and MMedBench, then pre-trains and evaluates multilingual medical language models using multiple data, prompting, and evaluation procedures.
- MMedC construction: MMedC contains over 25.5 billion medical tokens across English, Chinese, Japanese, French, Russian, and Spanish.Its sources include filtered multilingual-corpus content, medical textbooks, open-source medical websites, and existing small-scale multilingual medical corpora.
- MMedC construction: The corpus pipeline filters text using medical keyword count and keyword density thresholds, returning whether each sample is medical-related.The procedure defines unique medical keywords and density from keyword occurrences relative to text length.
- MMedC construction: MMedC supplements filtered web data with more than 20,000 medical textbooks and medical websites intended to cover important knowledge more comprehensively.Textbooks are processed with OCR, while websites compensate for the limited traceability and coverage of filtered CommonCrawl data.
- Auto-regressive training: Existing LLMs are further pre-trained autoregressively on MMedC using next-token prediction over medical token sequences.The optimization objective predicts each token from preceding tokens in the sequence.
- Benchmark and evaluation: MMedBench evaluates multilingual medical question answering and rationale generation across zero-shot, parameter-efficient fine-tuning, and full fine-tuning settings.The benchmark methodology also uses multilingual BERT embeddings with cosine similarity for semantic rationale evaluation.
- Benchmark and evaluation: The evaluation includes English medical datasets PubMedQA and six medicine-related MMLU subjects.PubMedQA uses its manually labeled subset for testing, while the selected MMLU subjects contain 1,089 questions.
5 Conclusion
The paper presents MMedC, MMedBench, and evaluations showing that further training on MMedC improves adaptation of general multilingual LLMs to medicine.
- Conclusion: MMedC contains 25.5B tokens across six languages, while MMedBench evaluates multilingual medical multiple-choice and rationale abilities.The study evaluates eleven existing LLMs under varied settings and releases MMed-Llama 3 as an open-source multilingual medical model.
- Conclusion: Further training on MMedC significantly fills the gap in adapting advanced general multilingual LLMs to the complex medical domain.The paper reports impressive performance for MMed-Llama 3 on English benchmarks as well.
6 Data Availability
The paper releases its MMedC dataset, MMedBench benchmark data, and leaderboard resources online under stated licensing terms.
- Data availability: MMedC and MMedBench are released on Hugging Face with CC BY-NC-SA licenses, and the MMedBench leaderboard is hosted on the project website.The paper states that source data are provided, with the cited resources available online.
9 Author Contributions
The paper describes prompts, model-comparison metadata, filtering parameters, and rationale-generation and evaluation instructions used in its methodology and experiments.
- Prompting: The methodology uses separate prompts for direct multiple-choice answers and answers accompanied by rationales.Fine-tuning uses both instruction types, while evaluation uses the direct-answer prompt for accuracy and the rationale prompt for rationale assessment.
- Rationale evaluation: GPT-4 rates rationale responses by ranking six assistants according to instruction following, helpfulness, relevance, accuracy, depth, creativity, and detail.The judging prompt instructs against position, name, and response-length biases.
- Rationale generation: GPT-4 generates rationale text by explaining why the ground-truth option is selected in the question’s language.The generation template requests a 100-word analysis using the question, options, answer identifier, and rationale fields.
- Model comparison: Supplementary Table 1 records model sizes, target languages, and pre-training or instruction-tuning data using a base-model-plus-data notation.The notation is described as “{model}+{data}”.
- Data filtering: Supplementary Table 2 reports stringent data-filtering parameters configured from the CulturaX dataset.The stated goal is to procure high-quality data.
C.2 Data Statistics
The supplementary statistics describe the composition of MMedC and MMedBench across languages. They organize corpus-source amounts and benchmark statistics by language.
- MMedC composition statistics report data amounts from different sources for each language.
- The reported corpus and benchmark statistics are presented in supplementary tables.
- The MMedBench statistics provide detailed composition information for different languages.
E Detailed Experiment Results
The detailed experiment results cover multilingual ablations and rationale evaluation. They examine accuracy and rationale-related outcomes across six languages.
- Ablation study: The ablation study reports detailed results across languages.
- Rationale evaluation: The experiments analyze rationale evaluation using results from supplementary tables.
- Ablation study: Ablation results include both accuracy and rationale evaluation across six languages.
E.2 Different Metrics on Rationale Evaluation
Rationale quality is analyzed with multiple automatic metrics, while supplementary material also documents model comparisons, filtering settings, corpus statistics, and benchmark statistics.
- Metric-based evaluation: The analysis examines scores assigned to rationales generated by various models using a range of evaluation metrics.
- Metric-based evaluation: Rationale scores are analyzed using N-gram BLEU, N-gram ROUGE, and BERT Score metrics under PEFT and full fine-tuning.BERT Score averages N-gram BERT scores as introduced in the main body.
- Supporting statistics: Supplementary tables summarize compared existing LLMs and parameter settings for filtering data from CulturaX.The filtering configuration prioritizes quality over quantity.
- Supporting statistics: Additional supplementary tables report MMedC corpus statistics and MMedBench statistics for different languages.
E.3 Detailed Results on Rating Scores
The supplementary results include GPT-4 and human ratings across six languages, alongside detailed rationale metrics and case analyses under different evaluation settings.
- Rating scores: GPT-4 and human rating scores are reported separately for six languages.
- Rationale metrics: Rationale evaluation tables cover BLEU-1 and ROUGE-1, N-gram BLEU, N-gram ROUGE, and BERT Score metrics.The BLEU-1/ROUGE-1 table identifies the values before and after the slash as BLEU-1 and ROUGE-1, respectively.
- Model results: The reported model rows include MMedLM and Llama 3 with paired metric values across multiple columns.
- Case analysis: Individual cases are analyzed first in zero-shot settings and then in full fine-tuning settings.
F.1 Zero-shot Setting
In zero-shot evaluation, 7B-scale models struggle to follow answer-format instructions consistently, making automatic accuracy calculation difficult. Small instruction changes can alter generated answers, motivating evaluation in fine-tuning settings.
- 7B-scale language models inconsistently follow zero-shot instructions, complicating automatic accuracy calculation.The same instruction can produce answers expressed in multiple ways.
- Models may produce varied answer formats despite receiving the instruction “Answer with the best option directly.”
- Example zero-shot outputs include direct choices, explanations, and diagnostic claims such as HIV infection.
- Changing the instruction to “Answer the letter of the option directly” may produce a different answer.
- Because zero-shot performance is difficult to evaluate fairly and automatically, the study turns to fine-tuning settings.
F.2 Full Fine-tuning Setting
The full fine-tuning comparison examines MMed-Llama 3 against Llama 3 across English and multilingual medical cases. MMed-Llama 3 selects correct answers more effectively, especially in multilingual cases, and applies medical knowledge to support precise diagnoses.
- MMed-Llama 3 is presented as Llama 3 augmented with domain-specific knowledge.
- MMed-Llama 3 demonstrates superior correct-answer selection compared with Llama 3, especially in multilingual cases.
- In an English case, MMed-Llama 3 identifies eosinophilic infiltration and diffuse parenchymal inflammation on renal biopsy.
- It uses those renal findings as indicative of tubulointerstitial nephritis, leading to a precise diagnosis.
- The examples cover end-of-life cancer care, benzodiazepine-related anterograde amnesia, renal deterioration, and a Spanish dermatological case.
- Supplementary cases include medical questions presented in English, Chinese, Japanese, French, Russian, and Spanish.