PaperlayerView exact revision on arXiv

Source-linked AI summary

HuatuoGPT-II, One-stage Training for Medical Adaption of LLMs

Junying Chen, Xidong Wang, Ke Ji, Anningzhe Gao, Feng Jiang, Shunian Chen, Hongbo Zhang, Dingjie Song, Wenya Xie, Chuyi Kong, Jianquan Li, Xiang Wan, Haizhou Li, Benyou Wang

arXiv:2311.09774v2cs.CLcs.AIcs.LG

TL;DR

Conventional two-stage domain adaptation faces differing objectives and data distributions. HuatuoGPT-II unifies domain and supervised data as instruction-output pairs and achieves state-of-the-art results across Chinese medicine benchmarks, especially in Traditional Chinese Medicine.

  • Problem

    The conventional two-stage protocol combines distinct optimization objectives and data distributions, creating dual shifts during domain adaptation.

    Source: §1 · p. 2

  • Method

    The one-stage protocol converts domain-specific pretraining data into instruction-output pairs and combines it with fine-tuning data using priority sampling.

    Source: §1 · p. 2

  • Results

    Across Chinese medicine benchmarks, HuatuoGPT-II reports state-of-the-art performance and surpasses ChatGPT and GPT-4 in some aspects, especially Traditional Chinese Medicine; experts recorded a 38% win rate and 38% ties against GPT-4.

    Source: §1 · pp. 2–3, §5 · p. 10

  • Takeaways & Limitations

    The experiments support the simplified protocol for Chinese medical adaptation, including Traditional Chinese Medicine.

    Source: §1 · p. 3, §5 · p. 10

    HuatuoGPT-II should not be used to provide medical advice because it can generate plausible but inaccurate or misleading information.

    Source: Ethic Statement · p. 10

Abstract

from arXiv · show

Adapting a language model into a specific domain, a.k.a `domain adaption', is a common practice when specialized knowledge, e.g. medicine, is not encapsulated in a general language model like Llama2. The challenge lies in the heterogeneity of data across the two training stages, as it varies in languages, genres, or formats. To tackle this and simplify the learning protocol, we propose to transform heterogeneous data, from the both pre-training and supervised stages, into a unified, simple input-output pair format. We validate the new protocol in the domains where proprietary LLMs like ChatGPT perform relatively poorly, such as Traditional Chinese Medicine. The developed model, HuatuoGPT-II, has shown state-of-the-art performance in Chinese medicine domain on a number of benchmarks, e.g. medical licensing exams. It even outperforms proprietary models like ChatGPT and GPT-4 in some aspects, especially in Traditional Chinese Medicine. Expert manual evaluations further validate HuatuoGPT-II's advantages over existing LLMs. Notably, HuatuoGPT-II was benchmarked in a fresh Chinese National Medical Licensing Examination where it achieved the best performance, showcasing not only its effectiveness but also its generalization capabilities.

1 Introduction

Domain adaptation must bridge specialized knowledge and heterogeneous training data, while the conventional two-stage process brings different objectives, distribution shifts, and added complexity. HuatuoGPT-II uses a unified one-stage protocol and achieves strong results in Chinese medical evaluations, including Traditional Chinese Medicine.

1 Introduction

  • The conventional two-stage approach injects domain knowledge through continued pre-training and activates instruction following through supervised fine-tuning, but its stages differ in objectives and data distributions.The resulting dual shifts and interdependent stages add optimization complexity and require manual hyperparameter selection.Source: §1 · p. 2
  • The protocol converts domain-specific pre-training data into instruction-output pairs and uses priority sampling to shift focus from domain knowledge toward downstream fine-tuning.Sampling initially emphasizes pre-training data, then progressively shifts toward fine-tuning data.Source: §1 · p. 2
  • For Chinese healthcare, diverse multilingual pre-training data are converted into consistent Chinese instructions, bridging the English-heavy data setting before priority sampling integrates pre-training and fine-tuning instructions.Source: §1 · p. 2
  • HuatuoGPT-II surpassed other models on the fresh 2023 Chinese National Pharmacist Licensure Examination, with a 38% win rate and 38% tie rate against GPT-4 in expert evaluations.The examination was untouched, addressing test-data leakage concerns.Source: §1 · pp. 2–3
  • Across a spectrum of evaluation methods, HuatuoGPT-II’s one-stage protocol proved more effective than the traditional two-stage training paradigm.Source: §1 · p. 3

2 Data Collection

The data collection section describes the medical pre-training corpus and the fine-tuning instruction set used for Chinese healthcare adaptation.

2 Data Collection

  • Domain data are divided into pre-training corpora and fine-tuning instructions, which supply the two data types used in the adaptation setup.Source: §2 · p. 3
  • The medical pre-training corpus comprises 1.1TB of Chinese and English material from encyclopedias, books, literature, and web sources, yielding 5,252,894 documents.Sources blend general corpora such as C4 with specialized sources such as PubMed.Source: §2 · p. 3
  • The fine-tuning set contains 142K real-world medical questions from Huatuo-26M, paired with outputs generated by GPT-4.The instructions are intended to support user interaction within the medical domain.Source: §2 · p. 3

3 One-stage Adaption

One-stage adaptation unifies continued pre-training and supervised fine-tuning in a single stage. It converts domain corpus text into instruction-output data and uses priority sampling to shift from domain knowledge toward instruction interaction.

3 One-stage Adaption

  • The protocol combines continued pre-training and supervised fine-tuning into a single adaptation stage.It consists of data unification followed by one-stage training.Source: §3 · p. 4

3.1 Data Unification

  • Corpus text is converted into instruction-output data by generating instructions from the text and responses from the corpus and instructions.The resulting format matches the single-turn supervised fine-tuning data.Source: §3.1 · p. 4

3.2 One-stage Training

  • Priority sampling integrates medical pre-training and fine-tuning instructions, favoring pre-training data before gradually shifting toward instruction interaction.Sampling probabilities change dynamically; higher β favors more sequential sampling, while lower β encourages mixing.Source: §3.2 · pp. 4–6

4 Experiments

Experiments compare HuatuoGPT-II with existing models and one-stage versus two-stage adaptation, then test priority-sampling settings. The reported results favor HuatuoGPT-II on medical evaluations and show better performance and training stability for one-stage adaptation.

4.1 Experimental settings

  • Evaluation covers four medical benchmarks, two general benchmarks' medical components, national medical exams, and real-world single- and multi-round questions, with Baichuan2 and medical and proprietary models as baselines.HuatuoGPT-II builds on Baichuan2-7B-Base and Baichuan2-13B-Base; ShareGPT data supplements single-round medical instructions for multi-round dialogue.Source: §4.2 · p. 6, §4.1 · p. 6, §4.3 · p. 8

4.2 Medical Benchmark

  • HuatuoGPT-II surpassed all open-source models and closely rivaled ERNIE Bot on national exams; in 2023, it scored 51.6 in TCM Pharmacy and ranked second in Pharmacy behind GPT-4.The 2023 Chinese National Pharmacist Licensure Examination had Pharmacy and Traditional Chinese Medicine Pharmacy tracks.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Medical Response Quality

  • GPT-4 automatic evaluation gave HuatuoGPT-II higher win rates than other models, including GPT-4, and four physicians also judged it consistently better than its peers.The authors note that its fine-tuning data came from GPT-4, while its medical corpus provided additional medical knowledge.Source: §4.3 · p. 8, Table 3 · p. 8

4.4 One-stage Vs. Two-stage Adaption

  • One-stage Adaption significantly outperformed Two-stage Adaption on all six datasets, by 5.3% to 23%, especially on single-round Q&A and multi-round conversations.The comparison is shown in Figure 6; its Win Rate uses GPT-4 automatic evaluation.Source: §4.4 · p. 9, Figure 6 · p. 9

4.5 The Relative Priority for Sampling

  • Priority sampling performed best near β = 2 and dropped at β = 0 or excessively high β; an Appendix G ablation also identified data unification as crucial.The priority-sampling comparison used the same HuatuoGPT-II (7B) configuration.Source: Figure 7 · p. 9, §4.5 · p. 10

5 Conclusion

The paper proposes one-stage domain adaptation, using LLM capabilities to align domain corpora with supervised fine-tuning data and priority sampling. HuatuoGPT-II performs strongly across Chinese medical benchmarks, while the experiments report an advantage over conventional two-stage adaptation.

5 Conclusion

  • The proposed one-stage method simplifies conventional two-stage domain adaptation and is intended to mitigate its associated challenges.The recipe aligns domain corpora with supervised fine-tuning data and uses priority sampling.Source: §5 · p. 10
  • HuatuoGPT-II achieves state-of-the-art performance across Chinese medicine benchmarks and surpasses ChatGPT and GPT-4 in some aspects, particularly Traditional Chinese Medicine.Source: §5 · p. 10
  • Experiments report a significant improvement for one-stage adaptation over the traditional two-stage approach.The authors frame the method as a potential framework for future LLM domain-adaptation work.Source: §5 · p. 10

Ethic Statement

The statement cautions against using HuatuoGPT-II for medical advice and describes its data sources and data-unification process. It also reviews domain-adaptation motivation and earlier Chinese medical LLM approaches.

Ethic Statement

  • HuatuoGPT-II should not be used for medical advice because LLMs can generate plausible but inaccurate or misleading information; its training sources include textbooks and literature, not patient-specific data.The authors say LLM-based rewriting of medical corpora aims to remove ethically questionable content.Source: Ethic Statement · p. 10

A.1 Domain adaption

  • Prior work reports that further domain training can sharply reduce performance despite benefits on fine-tuning and knowledge-probing tests; selected 10B datasets may rival larger models.Source: §A.1 · p. 15

A.2 Medical LLMs

B Supplementary Experiment

HuatuoGPT-II’s supplementary experiments report advantages across Chinese medical examinations, clinical diagnosis, and USMLE, while the paper details its corpus collection and data-unification process. The experiments also compare pre-training data with instruction-tuning data as sources of medical knowledge.

B Supplementary Experiment

  • HuatuoGPT-II (13B) surpassed open-source models and closely approached ERNIE Bot on the Chinese National Medical Licensing Examination.The table caption notes that the exams represent actual examination years and may be incomplete.Source: Table 4 · p. 15, B Supplementary Experiment · p. 16
  • HuatuoGPT-II outperformed all CMB-Clin counterparts except GPT-4, while its 7B version outperformed its 13B version.CMB-Clin evaluates clinical diagnosis using 74 textbook-derived complex, real-world cases and GPT-4 as evaluator.Source: B Supplementary Experiment · p. 16, Table 5 · p. 16
  • HuatuoGPT-II (13B) outperformed comparative open-source models on USMLE, though it remained behind ChatGPT.The passage identifies the exam’s English-centric nature as a constraint for a model designed primarily for Chinese medical contexts.Source: B Supplementary Experiment · p. 16, Table 6 · p. 17

C Domain Corpus Collection

  • The collection pipeline gathered 1.1TB of Chinese and English data from web corpora, books, encyclopedias, and medical literature, yielding 5,252K instruction data.Sources included C4, Wudao, textbooks, Chinese Medical Encyclopedia, PubMed, and Chinese literature.Source: C Domain Corpus Collection · pp. 17–18
  • The corpus pipeline filters medical content with a vocabulary dictionary, segments text, removes advertisements and low-quality text, and deduplicates semantically similar passages.The deduplication step uses sentence embeddings and dense retrieval.Source: C Domain Corpus Collection · pp. 17–18

D Example of Data Unification

E Pre-training Instruction: Knowledge Provision

  • Comparing equal additions of data, pre-training data improved medical benchmarks more than instruction-tuning data; the authors attribute the gains primarily to raw data.External LLMs are used to transform raw medical corpora into formats digestible to the trained model, while fine-tuning data mainly supports instruction following and conversation.Source: E Pre-training Instruction: Knowledge… · p. 19
  • The authors report that HuatuoGPT-II surpasses DISC-MedLLM on domain benchmarks, although both use ChatGPT or GPT-4 in training-data construction.Source: E Pre-training Instruction: Knowledge… · p. 19

F Data Unification without External LLMs

The authors test whether one-stage adaptation requires external LLMs for data unification. Using Baichuan2-7B-Chat for self-data-unification, their one-stage method still outperformed two-stage adaptation.

F Data Unification without External LLMs

G Ablation Study of Data Unification

The ablation reports that data unification is important for the one-stage method, and the section describes deviation checks, data ordering, and training settings. The checks combine lexical overlap and a trained detector, while training prioritizes pre-training data before fine-tuning data.

G Ablation Study of Data Unification

  • Without data unification, the single-stage method fell short of the two-stage method on the reduced-dataset ablation.The authors suggest that discrepancies between the two data stages make simple merging ineffective; the table compares performance with and without data unification.Source: G Ablation Study of Data Unification · p. 20, Table 11 · p. 20

H Deviate Detection

  • Generated responses are checked against source text using 1-gram Jaccard similarity or a human-annotated detector, then regenerated if they fail.The statistical check is suited to same-language cases; the trained model detector handles cases such as English text translated into Chinese responses.Source: H Deviate Detection · p. 21

I Priority Sampling Strategy

  • Training orders pre-training data before fine-tuning data, with within-corpus priority adjusted after medical-expert consultation to rank books above encyclopedias.The resulting source order is Web Corpus→Literature→Encyclopedia→Books.Source: I Priority Sampling Strategy · p. 21

J Training Detail

  • HuatuoGPT-II uses β=2, samples pre-training instructions at epoch 3 and fine-tuning instructions at epoch 1, and trains on fixed-length 4096-token sequences.Training used batch size 128, learning rate 1e-4, output-only loss, and eight Nvidia A100 cards; the 7B model took about four days.Source: J Training Detail · p. 22

K Benchmark Details

The appendix specifies the medical benchmarks, response evaluations, baselines, and supplementary experiments used to assess HuatuoGPT-II. It reports minimal impact from alternative SFT data and describes evaluations of model size and synthesized-question types without giving their detailed results here.

K Benchmark Details

  • The medical benchmark set includes MedQA, MedMCQA, CMB-Exam, and CMExam, alongside medically related portions of C-Eval and CMMLU.The described evaluations use the English MedQA test set, MedMCQA development set, and Chinese CMB-Exam; the general benchmarks are restricted to medical content.Source: K Benchmark Details · p. 22

L Details of HuatuoEval

  • HuatuoEval covers single-round patient questions from KUAKE-QIC and multi-round diagnostic cases from Med-dialog.The multi-round evaluation simulates a patient conversation and requires each model to respond twice.Source: L Details of HuatuoEval · p. 23

L.1 Automatic Evaluation

  • Automatic evaluation uses GPT-4 to compare paired model responses, with each data point judged in both interaction positions to mitigate position bias.The table caption identifies results from this automatic evaluation in Chinese medical scenarios.Source: Table 13 · p. 23, §L.1 · p. 24

L.2 Expert Evaluation

  • Licensed doctors compare anonymized, position-randomized response pairs using criteria covering relevance, clarity, accuracy, and humanistic quality.Figure 10 shows the expert-evaluation interface.Source: §L.2 · p. 25, Figure 10 · p. 26

M Baselines

  • Baselines include general chat models, Chinese medical models DISC-MedLLM and HuatuoGPT, and proprietary models such as ERNIE Bot, ChatGPT, and GPT-4.The described open-source general baselines include Baichuan2, Qwen, ChatGLM3, and Llama2.Source: M Baselines · pp. 26–27

N Distribution of Synthesized Questions

O Performance on Different Sizes of Models

P Other SFT Data

  • Using other LLMs to generate 5% of SFT data had minimal impact, which the authors interpret as evidence that SFT data does not provide primary knowledge.Table 17 is identified as reporting results on different SFT data.Source: P Other SFT Data · p. 27, Table 17 · p. 28

Q One Stage Training for Other Domains

The section tests one-stage training in English medical and law domains and reports improved performance across domains; it also identifies Chinese medical models used as baselines.

Q One Stage Training for Other Domains

R Other Medical Models

  • HuatuoGPT and DISC-MedLLM were selected as Chinese medical baselines based on their superior performance in the Chinese National Medical Licensing Examination.Source: R Other Medical Models · p. 28

S Case Study

Chinese medical case studies illustrate hallucinations by GPT-4 and Baichuan2-7B-Chat, while HuatuoGPT-II answers the cited drug questions accurately. The authors attribute hallucinations to gaps in medical knowledge and misconceptions in Chinese.

  • The authors attribute Chinese medical hallucinations to missing specific medical knowledge and misconceptions arising in Chinese.Source: S Case Study · p. 28
  • In the methoxyphenamine case, GPT-4 gives an irrelevant response, Baichuan2-7B-Chat is partly correct but hallucinates, and HuatuoGPT-II answers accurately and comprehensively.Source: S Case Study · p. 29, Table 21 · p. 30
  • In a Chinese question about OxyContin, GPT-4 erroneously associated it with Oscillococcinum, Baichuan2-7B-Chat referred to a nonexistent drug named Oscilloclasm, and HuatuoGPT-II correctly understood this as Oxycodone and provided accurate information.Source: S Case Study · p. 30
  • The authors argue that healthcare needs stronger domain-specific capabilities and say HuatuoGPT-II appears more adept at this during its current developmental phase.Source: S Case Study · p. 31
Loading 2311.09774v2…