PaperlayerView exact revision on arXiv

Source-linked AI summary

Baichuan-M1: Pushing the Medical Capability of Large Language Models

Bingning Wang, Haizhou Zhao, Huozhi Zhou, Liang Song, Mingyu Xu, Wei Cheng, Xiangrong Zeng, Yupeng Zhang, Yuqi Huo, Zecheng Wang, Zhengyun Zhao, Da Pan, Fei Kou, Fei Li, Fuzhong Chen, Guosheng Dong, Han Liu, Hongda Zhang, Jin He, Jinjie Yang, Kangxi Wu, Kegeng Wu, Lei Su, Linlin Niu, Linzhuang Sun, Mang Wang, Pengcheng Fan, Qianli Shen, Rihui Xin, Shunya Dang, Songchi Zhou, Weipeng Chen, Wenjing Luo, Xin Chen, Xin Men, Xionghai Lin, Xuezhen Dong, Yan Zhang, Yifei Duan, Yuyan Zhou, Zhi Ma, Zhiying Wu

arXiv:2502.12671v2cs.CL

TL;DR

Medical LLM development is constrained by limited high-quality domain-specific data and the complexity of medical knowledge. Baichuan-M1 is trained from scratch on 20 trillion tokens with a focus on medical expertise. Its instruct model surpasses Qwen2.5-72B-Instruct in medicine, while the reported gap to leading proprietary models is no longer significant.

  • Problem

    Medical LLM development faces limited high-quality domain-specific data and complex medical knowledge, while fine-tuning general models may not capture the depth needed for complex queries.

    Source: §1 · p. 2

  • Method

    Baichuan-M1 is trained from scratch on 20 trillion tokens, balancing general capabilities and medical expertise.

    Source: §1 · p. 2

  • Results

    Baichuan-M1-14B-Instruct surpasses Qwen2.5-72B-Instruct in medicine, while its gap to Claude-3.5-Sonnet and GPT-4o is no longer significant.

    Source: §6.3 · pp. 17–18

  • Takeaways & Limitations

    Open-sourced Baichuan-M1-14B provides a resource for further exploration and refinement of medical LLMs.

    Source: §7.2 · p. 19

    The authors identify rare-disease diagnosis and real-world clinical consultation as areas needing improvement.

    Source: §7.2 · p. 19

Abstract

from arXiv · show

The current generation of large language models (LLMs) is typically designed for broad, general-purpose applications, while domain-specific LLMs, especially in vertical fields like medicine, remain relatively scarce. In particular, the development of highly efficient and practical LLMs for the medical domain is challenging due to the complexity of medical knowledge and the limited availability of high-quality data. To bridge this gap, we introduce Baichuan-M1, a series of large language models specifically optimized for medical applications. Unlike traditional approaches that simply continue pretraining on existing models or apply post-training to a general base model, Baichuan-M1 is trained from scratch with a dedicated focus on enhancing medical capabilities. Our model is trained on 20 trillion tokens and incorporates a range of effective training methods that strike a balance between general capabilities and medical expertise. As a result, Baichuan-M1 not only performs strongly across general domains such as mathematics and coding but also excels in specialized medical fields. We have open-sourced Baichuan-M1-14B, a mini version of our model, which can be accessed through the following links.

1 Introduction

Medical LLM development faces a shortage of high-quality domain-specific data and the challenge of complex, specialized knowledge. Baichuan-M1 is trained from scratch for medicine, balancing medical expertise with general capabilities.

1 Introduction

  • High-quality domain-specific data is scarce, while medical knowledge is complex and existing fine-tuning approaches may lack depth for complex medical queries.Medical information is difficult to access and spans diverse structured and unstructured sources.Source: §1 · p. 2
  • Baichuan-M1 is trained from scratch on 20 trillion tokens, combining general and medical data to balance broad capabilities with medical expertise.The paper also describes an enhanced Transformer architecture and a progressive training process.Source: §1 · p. 2
  • The authors report that Baichuan-M1 handles general tasks and specialized medical applications, with higher accuracy and relevance on medical queries than general-purpose LLMs.The stated applications include diagnostic support, medical research, and treatment recommendations.Source: §1 · p. 2

2 Data

The data program combines a 20T-token multilingual general corpus with collected, filtered, and synthesized medical data. It uses controlled sampling and staged training to balance coverage, quality, and increasing medical complexity.

2.1 General Data

  • Global deduplication followed by upsampling according to document duplication counts significantly outperforms global deduplication alone in a 3B-model experiment.The strategy also gives better control over token count than single-bucket deduplication.Source: §2.1 · p. 3, Figure 2 · p. 4
  • Upsampling high-quality general data improves performance; for code, top 20% quality data sampled 5x performs better than no sampling or top 50% sampled 2x.Top 10% general data sampled 10x yields results comparable to top 33% sampled 3x; code comparisons maintain consistent code-data volume.Source: §2.1 · pp. 3–4, Figure 3 · p. 5, Figure 4 · p. 5
  • High-quality multilingual data mixed at 10% slightly enhances Chinese and English capabilities while adding multilingual capability and about 10% more tokens.The authors report that simply filtering low-quality multilingual data and mixing in a large amount could not preserve the model's original Chinese and English capabilities.Source: §2.1 · p. 4, Figure 5 · p. 6
  • Data is classified into 27 categories, with overrepresented web domains downsampled and underrepresented areas such as STEM upsampled.Large-scale data-ratio experiments are used to fit a strategy for performance across domains.Source: §2.1 · p. 4

2.2 Medical data

  • Medical data combines classified web corpora with expert-curated sources, and a three-stage process progressively increases medical-data concentration and complexity.Curated sources include papers, cases, textbooks, knowledge graphs, guidelines, encyclopedias, and online medical QA; the database spans over 1T tokens from over 200 sources.Source: §2.2 · p. 6, §2.2.1 · p. 6

2.2.1 Data source

  • Medical quality and value scores guide filtering and sampling, while expert-curated sources serve as seeds for synthetic data designed for accuracy and coverage.The synthesis emphasizes long Chain-of-Thought reasoning and applies medical-value reward checks, revising data that does not meet quality standards.Source: §2.2.1 · p. 7, §2.2.2 · pp. 7, 9

2.2.2 Synthetic Data

  • Synthetic medical QA covers long-tailed topics, while patient-case synthesis reconstructs expert reasoning from notes that often record decisions but omit the reasoning process.Long-tailed coverage includes biomedical statistics, numerical problems, rare diseases, and special populations.Source: §2.2.2 · p. 8
  • Synthetic-data pipelines vary by source, including entity-as-answer and contrastive questions from knowledge graphs and evidence-to-reasoning bridges from academic papers.For textbooks, encyclopedias, and guidelines, the pipeline extracts knowledge points, generates questions and unreferenced answers, then revises answers against source documents.Source: §2.2.2 · pp. 7–9

3 Model Architecture

Baichuan-M1 combines familiar transformer components with interleaved global and sliding-window attention, temporal short convolutions, and long-context rotary embeddings. In ablations, sliding-window attention improved short-context benchmarks without significantly affecting long-context benchmark performance.

3 Model Architecture

  • The overall structure also uses RMSNorm pre-normalization, SwishGlu feed-forward layers, rotary position embeddings, and a 256-dimensional head for global attention.The authors report that early experiments found the 256 head dimension beneficial for some benchmarks.Source: §3 · p. 9
  • Temporal short convolutions on attention keys and values are used to support in-context learning.The paper identifies this structure as beneficial for the model’s in-context learning ability.Source: §3 · p. 9, Figure 8 · p. 10
  • The model uses a rotary-embedding base of 1,000,000 for its 32k context length; the authors say a smaller base would lose long-distance retrieval ability.They contrast this with a base of 10,000 used by Llama2.Source: §3 · p. 10
  • Interleaved global and sliding-window attention is used to reduce inference cost and KV-cache use.Global attention has 2 heads with head dimension 256; sliding-window attention has 8 heads with head dimension 128.Source: §3 · p. 10
  • Sliding-window attention improved short-context benchmark performance without significantly affecting long-context benchmark performance.The ablation used a 1.5B-parameter model trained on 200B tokens.Source: §3 · p. 10, Table 2 · p. 10

4 Training Progress

Baichuan-M1 uses a combined general-purpose and medical tokenizer, then pretrains through a curriculum that progressively increases data difficulty and medical focus. Training details include adaptive gradient clipping for early stability and staged changes to context length and learning rate.

4.1 Tokenizer

  • Baichuan-M1 combines a general-purpose tokenizer with a specialized medical tokenizer into a 133,120-token vocabulary.The combination is intended to balance general tokenization efficiency with medical-term granularity.Source: §4.1 · pp. 10–11

4.2 Training details

  • Pretraining follows three curriculum stages: easier data first, then more hard samples and medical data, followed by complex, application-specific data for downstream alignment.The first stage does not significantly increase the medical-data proportion; the second builds medical capabilities on general capabilities.Source: §4.2 · p. 11
  • Baichuan-M1-14B training used AdamW, a peak learning rate of 4e-4, and 16M batch size, with context length increasing from 8K to 32K during annealing.Training progressed through 12T tokens, another 6T tokens with more high-quality and serious medical data, then 2T annealing tokens before SFT.Source: §4.2 · p. 11
  • Adaptive gradient clipping was used early in training to reduce the influence of unusually large gradients caused by special data and improve training stability.The authors report testing its stability benefit on a 3B-parameter model; Figure 10 shows improved initial-loss stability.Source: §4.2 · pp. 11–12, Figure 10 · p. 12

5 Alignments

Baichuan-M1 alignment combines general and medical instruction tuning with reward-based reinforcement learning. Medical SFT spans clinical knowledge, language, reasoning, long-context, and safety tasks; the subsequent policy stages target reasoning paths, preferences, and user-aligned generation.

5.1.1 General SFT data

  • General SFT data was iteratively refined and includes task-focused mathematics and coding competition examples to emphasize reasoning skills.The instruction-tuning dataset combines general and medical instruction data.Source: §5.1 · p. 12, §5.1.1 · p. 13

5.1.2 Medical SFT data

  • Medical SFT covers five categories: medical knowledge, language understanding, reasoning, long context, and safety.Long-context examples include question answering, summarization, rewriting, and evidence extraction; safety data uses coverage labels and behavior guidelines.Source: §5.1.2 · pp. 13–14
  • Medical reasoning data includes case summaries and differential diagnosis, while treatment recommendations draw on guidelines and discuss strategy pros and cons.The authors describe an iterative review process that refines conclusions using clinical best practices and feedback loops.Source: §5.1.2 · p. 13

Answer Quality

  • Answer-quality data uses real-world cases such as MIMIC and PMC-Patient to generate questions, then combines candidate generation with human expert review.Experts review and refine candidates for safety and accuracy while preserving critical medical context.Source: Answer Quality · p. 14

5.2.1 Reward Model

  • The reward process combines rule-based feedback for verifiable questions with a model-based reward model for questions with uncertain answers.The model-based reward model is trained from a Baichuan-M1 checkpoint using preference data designed to impose expert priors and reduce reward-hacking risk.Source: §5.2.1 · p. 14

5.2.2 Training Policies

  • The three policy stages are ELO, token-level DPO, and PPO, progressively refining reasoning paths, preferences, and generation using reward-model feedback.ELO optimizes ground-truth answer likelihood without external reward signals; TDPO applies preference constraints at token level, and PPO uses reward-model feedback.Source: §5.2.2 · p. 15, Figure 11 · p. 15

6 Evaluations

Baichuan-M1 is evaluated on open and private datasets spanning medical fundamentals, examinations, and practice, with generated free-text answers processed or scored using Qwen2.5-72B-Instruct. The reported comparison places its medical performance above Qwen2.5-72B-Instruct, while it remains behind proprietary models with a gap described as no longer significant.

6.1 Benchmarks

  • Evaluation covers open-source and private datasets across medical fundamentals, examinations, and medical practice.The benchmarks include tasks such as MedNLI and MedCalc, medical exams, and consultation or diagnosis settings; at most 1,000 samples were randomly selected per benchmark.Source: §6.1 · p. 16

6.3 Results

  • Baichuan-M1-14B-Instruct surpasses Qwen2.5-72B-Instruct in medicine, while it still lags Claude-3.5-Sonnet and GPT-4o, with the gap described as no longer significant.Table 3 presents results against strong baselines.Source: Table 3 · p. 17, §6.3 · pp. 17–18

6.2 Response and Scoring

  • Responses are free text, with Qwen2.5-72B-Instruct used for post-processing or scoring; multiple-choice scores are based on extracted options compared with correct answers.Scores for other benchmarks are obtained using prompts, and benchmark scores are normalized.Source: §6.2 · p. 17

6.3 Results

  • The paper also evaluates Baichuan-M1-14B-Base on code generation and mathematics using EvalPlus, Bigcodebench, MATH, and CMATH.The cited passage describes evaluation tasks but does not provide their results.Source: §6.3 · p. 18, Table 4 · p. 18

7 Discussion and Conclusion

The discussion questions whether continued pretraining can improve medical capability without sacrificing general performance. It presents Baichuan-M1’s from-scratch training as effective across medical applications while noting areas needing improvement.

7.1 Continue Pre-training or Training from Scratch?

  • Continued training on well-trained, especially annealed, base models made it difficult in preliminary experiments to improve vertical capabilities without sacrificing general capabilities.The discussion suggests simply increasing domain-specific data or adjusting the learning rate may be ineffective for improving medical capabilities.Source: §7.1 · p. 18

7.2 Conclusion and Future Work

  • Baichuan-M1 is presented as trained from scratch for medical expertise and robust across applications from diagnostics to treatment recommendations.The conclusion contrasts this approach with fine-tuning general-purpose models.Source: §7.2 · p. 18
  • Its training integrates medical literature, clinical data, and expert-curated sources and uses synthetic data with a finely tuned pre-training process.The passage identifies these sources and methods as strengths of the model.Source: §7.2 · p. 18
  • Baichuan-M1 is described as balancing general and medical capabilities, excelling at common language tasks and specialized medical queries.The conclusion also notes room for improvement, especially in rare disease diagnosis and real-world clinical consultation.Source: §7.2 · p. 19
  • Open-sourcing Baichuan-M1-14B is presented as a resource for further research and refinement of medical LLMs.The conclusion describes the model as a promising step toward meeting medicine’s precision demands.Source: §7.2 · p. 19

A Contributor

The passage lists named contributors in two groups.

A.2 Core Contributors

  • Core contributors named here are Bingning Wang, Haizhou Zhao, Huozhi Zhou, Liang Song, Mingyu Xu, and Wei Cheng.Source: §A.2 · p. 26

A.3 Contributors

  • Additional contributors named here include Da Pan, Fei Kou, Fei Li, Fuzhong Chen, Guosheng Dong, Han Liu, Hongda Zhang, Jin He, Jinjie Yang, Kangxi Wu, Kegeng Wu, Lei Su, Linlin Niu, Linzhuang Sun, Mang Wang, and Pengcheng Fan.Source: §A.3 · p. 26

B Examples of synthetic data

This section presents synthetic-data examples, including medical questions and answers about surgical choices, disease mechanisms, and vitamin K.

B Examples of synthetic data

B.1 Example 1

  • For a lower descending thoracic aortic aneurysm requiring an invasive procedure, the example identifies open surgical repair as appropriate, unlike the less-invasive TEVAR option.The example says TEVAR is used for thoracic aneurysms but does not meet the stated invasive-method criterion.Source: §B.1 · p. 27

B.2 Example 2

  • The Wilson’s disease example identifies abnormal copper accumulation in brain regions, free-radical generation, and neuronal damage as the pathological process.Its summary says the neurological symptoms, systemic findings, and imaging align with Wilson’s disease.Source: §B.2 · p. 28

B.3 Example 3

  • The vitamin K example explains that r-carboxylation enables clotting factors to bind calcium, while deficiency reduces their activity and impairs clotting.Source: §B.3 · p. 28
  • Listed deficiency problems include bleeding and hemorrhagic disease of the newborn, with newborn risk linked to limited intestinal bacterial synthesis and low vitamin K in breast milk.Source: §B.3 · pp. 28–29
  • The example also links vitamin K deficiency to decreased bone density, fracture risk, increased vascular calcification, and cardiovascular disease risk.Source: §B.3 · p. 29

B.4 Example 4

The example calculates the sample size for comparing two treatments’ mean blood-flow increases using a two-sample t-test. Under the stated inputs, it gives 59 animals per group, or 118 total.

B.4 Example 4

  • The calculation uses the sample-size formula for comparing two means in a two-sample t-test.Source: §B.4 · p. 29
  • The setup assumes equal standard deviations of 1.0 ml/min and uses treatment means of 1.8 and 2.4 ml/min, with α = 0.05 and β = 0.10.The absolute mean difference used is 0.6 ml/min.Source: §B.4 · p. 29
  • 59 animals per group are needed, for 118 animals total, to detect the stated treatment difference at the given significance level and power.Source: §B.4 · p. 30

C Tokenizer Details

Baichuan-M1 combines a SentencePiece base tokenizer with an additional multilingual and medical BPE tokenizer. The section lists tokenizer rules and presents tokenization-efficiency comparisons across models and languages.

C Tokenizer Details

  • The tokenizer combines a SentencePiece base general tokenizer with an additional multilingual and medical BPE tokenizer.The two tokenizers are then merged.Source: C Tokenizer Details · p. 30
  • Input normalization is disabled, whitespace-only pieces are retained, numbers are split into individual digits, and character coverage is 0.9999 with rare characters falling back to UTF-8 bytes.Source: C Tokenizer Details · p. 30
  • Figure 12 compares tokenization efficiency across models and languages, with lower values indicating better efficiency.Source: Figure 12 · p. 30, C Tokenizer Details · p. 30

D Evaluation Prompts and Examples

Appendix D lists prompts used for response generation and benchmark scoring, alongside examples of Baichuan-M1-14B-Instruction.

D Evaluation Prompts and Examples

Loading 2502.12671v2…