Towards Building Multilingual Language Model for Medicine

1 Introduction

• MMedC contains over 25.5 billion medical-related tokens across English, Chinese, Japanese, French, Russian, and Spanish.It combines automatically filtered multilingual content, curated textbooks, open-source medical websites, and existing small medical corpora; the authors describe it as the first corpus specifically focused on multilingual medicine.Source: §1 · pp. 1–2, Figure 1 · p. 2
• MMedBench provides 53,566 medical QA pairs across six languages and 21 fields, with multiple-choice answers and accompanying rationales.It has 45,048 training pairs and 8,518 test pairs; a subset of 1,136 test pairs has manually verified rationales for reasoning evaluation.Source: §1 · pp. 2–3, Figure 1 · p. 2
• The evaluation compares multilingual LLMs and MMedC-trained models in zero-shot, PEFT, and full fine-tuning settings, using automated metrics and human ratings for rationale quality.The benchmark evaluates multiple-choice accuracy and rationale generation, including zero-shot and fine-tuning settings.Source: §1 · pp. 1, 3
• MMed-Llama 3 performs best on multilingual and English-only benchmarks, while models further trained on MMedC consistently improve.Figure 1 also reports the best MMedBench multiple-choice accuracy among existing open-source LLMs for MMed-Llama 3.Source: Figure 1 · p. 2, §1 · p. 3

Data Statistics

• MMedC contains over 25.5B tokens across six languages, with English at 42% and Russian at 7%.It draws mainly on filtered medical content, textbooks, medical websites, and existing small-scale corpora; the Russian share is approximately 2B tokens.Source: Data Statistics · p. 3
• MMedBench includes questions with multiple correct options and rationale sections averaging 200 tokens, spanning foundational and specialized medical topics.Internal Medicine and Biochemistry are among the more emphasized topics.Source: Data Statistics · p. 4

Evaluation on MMedBench

• MMed-Llama 3 reached average accuracy 67.75 under full fine-tuning, above Llama 3’s 62.79 and near GPT-4’s 74.27.The comparison is on MMedBench across six languages.Source: Evaluation on MMedBench · p. 5, Table 1 · p. 6
• MMed-Llama 3 earned the highest rationale ratings: 4.10 from human evaluators and 4.73 from GPT-4.Its GPT-4 rating exceeded the second-best model, InternLM 2, by 0.89 points.Source: Evaluation on MMedBench · pp. 7–8, Figure 4 · p. 8
• GPT-4 ratings had the highest correlation with human judgments (τ = 0.660), while BERT Score was the most reliable absolute metric (τ = 0.538).The authors recommend BERT Score for assessing rationale ability in newly introduced models.Source: Evaluation on MMedBench · p. 8, Figure 4 · p. 8

Evaluation on Public English Benchmarks

• MMed-Llama 3 achieved state-of-the-art performance on English benchmarks, with reported gains of 4.5%, 4.3%, and 2.2% on MedQA, MedMCQA, and PubMedQA.On MMLU-Medical, it scored 72.59 versus GPT-3.5’s 67.69.Source: Evaluation on Public English Benchmarks · p. 8, Table 3 · p. 9

Ablation Studies on Data Composition

• Full multilingual MMedC training raised accuracy to 64.40 and BLEU-1/ROUGE-1 by 0.48/0.54; adding US-Data raised accuracy to 67.75.English-only training did not improve overall accuracy; the authors suggest English overfitting, while multilingual training largely alleviated the issue. Comprehensive rationales raised accuracy from 58.72 to 62.79, and adding US-Data further increased BLEU-1/ROUGE-1 by 0.29/0.16.Source: Ablation Studies on Data Composition · p. 9

Experimental Results

• All three MMedC-adapted models improved over their InternLM, InternLM 2, and Llama 3 baselines, supporting specialized corpus training for multilingual medical contexts.The authors interpret the gains as evidence that existing pretraining corpora have gaps in multilingual medical contexts.Source: §3 · p. 9
• High-quality multilingual data and lower-quality general-corpus data both improved model performance, supporting diverse data sources in MMedC.The reported findings are presented as an ablation study on MMedB; the supplied passages do not give its individual score values.Source: §3 · pp. 9–10, Table 4 · p. 10
• Rationale-based fine-tuning raised MMedBench test accuracy by 2.33% for InternLM, 2.42% for InternLM 2, and 4.07% for Llama 3.Combining answers with rationales also enabled rationale sentence output during supervised fine-tuning.Source: §3 · p. 10, Table 4 · p. 10
• Later, stronger backbones generally performed better on multilingual medical QA: Llama 3 outperformed Llama 2 and Mistral, and MMed-Llama 3 followed MMedLM and MMedLM 2.The authors attribute this pattern to broader multilingual exposure in later models’ pretraining and recommend building open-source medical datasets to leverage general LLM improvements.Source: §3 · p. 10

Research Impacts

• Multilingual LLMs could expand GMAI’s use of worldwide data to improve multimodal training and representations, and enrich non-English medical knowledge bases for retrieval-augmented generation.The discussion frames multilingual retrieval as a potential response to medical hallucinations and the English focus of existing retrieval-augmented efforts.Source: §3 · pp. 10–11

Clinical Impacts

• In clinical practice, multilingual medical LLMs could support patient-provider communication across language barriers, address cultural and legal nuances, and build trust.The discussion also identifies multilingual medical education as a use case, particularly where educators or resources are scarce, to support training and care quality.Source: §3 · p. 11

Potential Limitations

• The authors identify web-crawl bias, underdeveloped architectural explainability, incomplete language coverage, and 8B-scale compute constraints as limitations.They propose stronger bias controls, adding languages such as German and Arabic and augmenting low-resource-language data, and exploring larger retrieval-augmented models to potentially improve results and alleviate hallucinations.Source: §3 · p. 11

4.1 Large-scale Multilingual Medical Corpus

• MMedC contains over 25B tokens in six languages, combining medical-filtered general-language text, textbooks, websites, and existing multilingual medical corpora.The six languages are English, Chinese, Japanese, French, Russian, and Spanish.Source: §4.1 · p. 12
• Corpus filtering uses unique medical-keyword count and keyword density thresholds; manual checks found an average of 98 of 100 sampled sentences per language were medical-related.The rule-based pipeline uses 200 medically relevant terms per language and applies segmentation before matching for space-delimited text.Source: §4.1 · pp. 12–13
• Textbooks were OCR-processed, while websites added professional, general, and timely medical information; the corpus also incorporates Wikipedia, Baidu Baike, and UFAL Medical Corpus.Textbook data include 4B English, 1.1B Chinese, 0.4B Russian, and 0.3B French tokens; website data include Japanese, Spanish, and French material.Source: §4.1 · p. 13

4.2 Auto-regressive Training on MMedC

• Existing LLMs are further pre-trained on MMedC with an auto-regressive next-token prediction objective.The objective follows the GPT loss formulation.Source: §4.2 · p. 13

4.3 Comprehensive Multilingual Medical Benchmark

• MMedBench aggregates medical multiple-choice QA across six languages, totaling 53,566 pairs, with 45,048 for training and 8,518 for testing.For datasets without official train-test divisions, the authors use an 8:1:1 train-validation-test split.Source: §4.3 · pp. 13–14
• GPT-4 generated rationales for questions, and human review found 94.7% of sampled rationales met quality criteria, yielding 1,136 verified samples for rationale evaluation.Reviewers checked agreement with the correct answer and whether explanations articulated reasoning rather than merely repeating the answer.Source: Figure 5 · p. 14, §4.3 · pp. 14–15
• Evaluation covered zero-shot, LoRA-based parameter-efficient fine-tuning, and full fine-tuning; GPT-4 was evaluated zero-shot, while BioMistral [23] is an open-source multilingual biomedical baseline and Gemini-1.0 pro [40] is a general multimodal model.Source: §4.3 · pp. 15–16
• Assessment combines accuracy and rationale similarity, including BLEU, ROUGE, and multilingual BERT-score, with human rankings of rationale answer correctness and reasoning ability.Human ratings compare six models on 50 cases per language.Source: §4.3 · p. 17
• Human, BLEU, and GPT-4 rankings were converted to reverse rank scores from 6 for first place to 1 for last.GPT-4 ranked model outputs using instructions; BLEU models were ranked by absolute metrics.Source: §4.3 · p. 18

4.4 English Benchmark Evaluation

• English evaluation used MedQA [17], labeled PubMedQA PQA-L [17], official MedMCQA test, and six medicine-related MMLU subjects [12].PQA-L was chosen for comparability with prior work; the six MMLU subjects comprise 1,089 questions. Since instruction-tuning data varied across models, comparisons used unexposed test sets, and MMed-Llama 3 received PMC-LLaMA English instruction data.Source: §4.4 · p. 18

4.5 Implementation Details

• Autoregressive training used 2,048-token chunks with 512-token overlap; training and fine-tuning used FSDP, BF16, and gradient checkpointing.InternLM and BLOOM used global batches of 512 and learning rates of 2e-5 and 8e-6, respectively; both were trained on eight A100 GPUs, with the 7B model trained for 20k iterations over about 20 days. Fine-tuning used a global batch of 128 and learning rate of 1e-6; LoRA used rank 16.Source: §4.5 · pp. 18–19

5 Conclusion

• The authors describe MMed-Llama 3 as, to their knowledge, the first and strongest multilingual medical language model, with impressive performance on English benchmarks.Source: §5 · p. 19
• MMedC contains 25.5B medical tokens across six main languages, while MMedBench covers six primary languages.Source: §5 · p. 19
• The authors evaluated eleven existing LLMs on multiple-choice question answering and rationale generation across various settings, finding that further training on MMedC helps adapt general multilingual LLMs to medicine.Source: §5 · p. 19

6 Data Availability

• MMedC Dataset and MMedBench Benchmark Data are released under CC BY-NC-SA, and the MMedBench Leaderboard is available online.The dataset links are https://huggingface.co/datasets/Henrychur/MMedC and https://huggingface.co/datasets/Henrychur/MMedBench; the leaderboard is at https://henrychur.github.io/MultilingualMedQA/.Source: §6 · p. 19

9 Author Contributions

• All listed authors meet ICMJE’s four criteria; Pengcheng Qiu and Chaoyi Wu contributed equally, and Weidi Xie is the corresponding author.The passage assigns conception or design contributions to all listed authors, acquisition, analysis, or interpretation to Qiu and Wu, drafting to Qiu and Wu, and critical review to the other listed authors; all approved the published version and accepted accountability.Source: §9 · p. 23

A Used Prompts

• The prompt appendix specifies zero-shot, fine-tuning, rationale-ranking, and rationale-generation instructions for the study.The appendix introduces the prompts used in the methodology and experiments.Source: A Used Prompts · p. 24, §A.0.1 · p. 24, §A.0.2 · p. 24

A.0.1 Zero-shot prompt

• For GPT-3.5 and GPT-4, zero-shot prompts request either only an option letter or a step-by-step rationale in the question’s language followed by the answer.The rationale format marks the end of the reason with [End].Source: §A.0.1 · p. 24

A.0.2 Fine-tuning Prompts

• Fine-tuning used separate prompts for direct answers and rationales, optimized each sample twice per epoch, and assigned the prompts to accuracy and rationale evaluation, respectively.The rationale prompt requests step-by-step reasoning in the question’s language before the answer index.Source: §A.0.2 · p. 24

A.0.3 Ranking Prompt

• The GPT-4 ranking prompt asks an impartial judge to order six assistants by response quality, considering helpfulness, relevance, accuracy, depth, creativity, and detail while avoiding position and length biases.The required output is the model ordering from high to low.Source: §A.0.3 · p. 24

A.1 Rationale Generation Prompt

• GPT-4 rationale generation prompts request a 100-word analysis in the question’s language using the question, options, and ground-truth answer.The template uses placeholders for the language, question stem, options, and answer.Source: §A.1 · p. 25

B Baseline Summarizing

• The baseline summary lists model size, target languages, and training data, distinguishing pre-training from instruction-tuning data and denoting additions as base model plus data.The summary covers the models compared in Supplementary Table 1.Source: B Baseline Summarizing · p. 25

C.1 Settings for Filtering Content

• Filtering parameters, based on CulturaX, use a relatively stringent criterion intended to procure high-quality data.The settings are presented in Supplementary Table 2.Source: §C.1 · p. 25

C.2 Data Statistics

• Supplementary Table 3 details MMedC data amounts from different sources for each language.Source: §C.2 · p. 25

D MMedBench Details

• Supplementary Table 4 provides MMedBench composition statistics across different languages.Source: D MMedBench Details · p. 25

E Detailed Experiment Results

• The section presents ablation results across languages and evaluates rationales using different metrics.Source: E Detailed Experiment Results · p. 25

E.1 Detailed Accuracy on Ablation Study

• The ablation study reports accuracy and rationale evaluation across six languages in Supplementary Tables 5 and 6.Source: §E.1 · p. 25

• Supplementary Tables 7 and 8 report each model’s N-gram BLEU performance under PEFT and full fine-tuning, respectively.Source: §E.2 · p. 25
• Supplementary Tables 9 and 10 present each model’s scores under N-gram ROUGE metrics.Source: §E.2 · p. 25
• Supplementary Tables 11 and 12 show BERT scores calculated by averaging N-gram BERT scores, as introduced in the main body.Source: §E.2 · p. 25

E.3 Detailed Results on Rating Scores

• Supplementary Table 5 presents accuracy across languages for the ablation study.Source: Table 5 · p. 28
• Supplementary Table 6 evaluates MMedB rationales using BLEU-1 and ROUGE-1, while Tables 7 and 8 report n-gram BLEU evaluations for PEFT and full fine-tuning.Source: Table 6 · p. 28, Table 7 · p. 29, Table 8 · p. 30
• Supplementary Tables 9 and 10 report n-gram ROUGE rationale evaluations for models using PEFT and full fine-tuning, respectively.Source: Table 9 · p. 31, Table 10 · p. 32
• Supplementary Tables 11 and 12 report BERT Score rationale evaluations for models using PEFT and full fine-tuning, respectively.Source: Table 11 · p. 32, Table 12 · p. 33
• Supplementary Tables 13 and 14 report detailed human and GPT-4 rating scores, respectively, with scores assigned quantitatively in reverse order of ranking.The preceding text specifies that the ratings cover six languages.Source: §E.3 · p. 27, Table 13 · p. 33, Table 14 · p. 33

F Case Study

• The case study analyzes individual outputs across languages, first in a zero-shot setting and then in a full fine-tuning setting.Source: F Case Study · p. 34

F.1 Zero-shot Setting

• 7B-scale models inconsistently follow instructions and express answers in varied formats, making automatic accuracy calculation difficult.Examples include direct answers, option letters, and answers with explanations.Source: §F.1 · p. 34
• Changing the instruction from asking for the best option to asking for the option letter may lead the model to give a different answer.Source: §F.1 · p. 34
• Because fair automatic evaluation in the zero-shot setting is difficult, the authors turn to fine-tuning settings.Source: §F.1 · p. 34

F.2 Full Fine-tuning Setting

• MMed-Llama 3 selects correct answers more effectively, especially in multilingual cases, and gives more adequate reasons than Llama 3.In an English case, it used renal-biopsy findings to identify tubulointerstitial nephritis and reach a precise diagnosis.Source: §F.2 · p. 34