Huatuo-26M, a Large-scale Chinese Medical QA Dataset

• 26,504,088 QA pairs make Huatuo-26M the largest Chinese medical QA dataset, expanding existing medical QA resources by more than two orders of magnitude.Source: §1 · pp. 1–2
• Existing retrieval and generation methods remain challenging to apply in medical QA, with benchmarked models performing below expected levels.The benchmark covers BM25, DeepCT, DPR, GPT2, and T5.Source: §1 · p. 1

Use case I: Transfer for other QA dataset

• Models trained on Huatuo-26M achieve competitive performance on cMedQA2 and webMedQA in few- or zero-shot settings.Source: §1 · p. 2

Use case II: As an external knowledge for RAG

• Using Huatuo-26M as external knowledge for RAG greatly improves generated-text quality on cMedQA2 and webMedQA.Source: §1 · p. 2

Use case III: As a pre-trained corpus

• Continued pre-training on Huatuo-26M largely improves pre-trained models’ performance on biomedical tasks.The paper applies QA-pair pre-training to BERT and RoBERTa.Source: §1 · p. 2

2 Dataset Creation

• Huatuo-26M aggregates medical QA from online consultation records, medical encyclopedias, and structured medical knowledge bases.Source: §2 · p. 2, §2.1 · p. 2, §2.3 · p. 3

2.1 Online Medical Consultation Records

• Online consultation processing yields 25,341,578 deduplicated QA pairs after removing special-character and repeated pairs.The raw crawl contains 31,677,604 QA pairs before filtering.Source: §2.1 · pp. 2–3

2.2 Online Medical Encyclopedia

• Encyclopedia articles are split into title-paragraph pairs and converted into questions with manually designed templates covering medical topics such as diagnosis and treatment.Source: §2.2 · p. 3

2.3 Online Medical Knowledge Bases

• Three knowledge graphs are cleaned, merged into 43 categories, and transformed into 798,444 QA pairs using manually designed templates.Templates query either an object entity from a subject entity or an entity attribute.Source: §2.3 · p. 3

• Huatuo-26M contains mostly online consultation QA pairs, with average question and answer lengths of 44.6 and 120.7, respectively.Questions and answers vary from long consultation and encyclopedia texts to shorter knowledge-base entries.Source: §3 · p. 3
• Questions are colloquial and patient-contextual, while doctor answers are professional, targeted, and semantically continuous.Source: §3 · p. 3

Questions are diverse

• Heuristic analysis finds many questions concern newborns, pregnant people, and children, with substantial consultation on behalf of elderly parents.Source: §3 · p. 4, Figure 1 · p. 4

Significant topics in Huatuo-26M

• Consultation records cover common, chronic, and some complex diseases, whereas encyclopedia and knowledge-base sources emphasize different specialties and more specialized diagnosis and treatment procedures.Consultation answers focus on medicines, lifestyle precautions, and hospital referrals; knowledge-base answers include procedures such as MRI, CT, and puncture.Source: §3 · p. 4

4 Benchmarks

• The benchmark covers answer retrieval and generation on Huatuo-26M using mainstream sparse, dense, and autoregressive methods.Source: §4 · p. 4, §4.1 · p. 4

4.1.1 Baselines and Experimental Settings

• Retrieval ranks the top 1000 answers from training and test pools, using 90% training data for encyclopedias and knowledge bases and 99% for consultant records or all categories.Source: §4.1.1 · p. 4
• The retrieval baselines are BM25, DeepCT, and DPR, evaluated with Recall@k and MRR@10.BM25 uses character-based indexes, DeepCT predicts context-aware term weights, and DPR uses a dual encoder with batch negatives.Source: §4.1.1 · pp. 4–5

4.1.2 Results

• DPR and DeepCT outperform BM25 in retrieval, and DPR is generally stronger than DeepCT across the evaluated settings.Recall is relatively low for consultant records because the 26M-answer candidate pool makes desired documents difficult to retrieve, although top-ranked answers can remain informative.Source: §4.1.2 · p. 5, Table 4 · p. 5
• Retrieval models generally perform poorly on knowledge-base QA because concise questions require understanding medical entities and their relationships.The authors characterize medical-domain knowledge representation in pretrained models as especially challenging.Source: §4.1.2 · p. 5

4.2.1 Baselines and Experimental Settings

• Generation fine-tunes T5 and GPT2 on Huatuo-26M and compares them with raw models using BLEU, ROUGE, GLEU, and Distinct metrics.Source: §4.2 · p. 5, §4.2.1 · pp. 5–6

4.2.2 Results

• Fine-tuned T5 and GPT2 significantly outperform their raw versions, with fine-tuned T5 achieving the best results across all generation indicators.The benchmark reports relatively weak absolute generation scores, partly because long medical answers are difficult to match exactly.Source: §4.2.2 · p. 6, Table 5 · p. 6

5 Applications

• The applications cover transfer to other QA datasets, external knowledge for RAG, and continued pre-training of existing language models.The application section organizes these uses as Sections 5.1, 5.2, and 5.3.Source: §5 · p. 6, §5.1 · p. 6, §5.2 · p. 7

5.3.2 Experimental Data

• The experiments use cMedQA2 and webMedQA for transfer and RAG, and CBLUE for evaluating continued pre-training.The RAG setup retrieves a top-ranked QA pair with a trained Chinese DPR model and supplies it to T5.Source: §5.1 · p. 6, §5.2 · p. 7, §5.3.2 · p. 8

5.1 Transfer for Other QA Dataset

• 25.42 and 22.73 absolute points improve raw T5 models on cMedQA2 and webMedQA, respectively, through Huatuo-26M training.A Huatuo-26M-trained T5 model also outperforms cMedQA2-trained T5 on cMedQA2 BLEU-1 without seeing that dataset.Source: §5.1 · p. 6

5.2 As an External Knowledge

• RAG improves text generation on cMedQA2, while additional Huatuo-26M pre-training adds 3 absolute points there but decreases webMedQA performance by nearly 6 points.The differing effects may depend on the characteristics of the target datasets.Source: §5.2 · p. 7

5.3 As a Pre-trained Corpus

• Huatuo-26M continued pre-training significantly improves BERT and RoBERTa performance on the CBLUE benchmark.CBLUE covers Chinese medical named-entity recognition, information extraction, and sentence or sentence-pair classification.Source: §5.3.3 · p. 8, §5.3.2 · p. 8

5.3.3 Results

• RoBERTa-base trained on Huatuo-26M scores 0.8 percentage points above RoBERTa-base and 0.5 above MC-BERT-base on average CBLUE performance.The trained 12-layer RoBERTa also outperforms the 24-layer RoBERTa model in average score.Source: §5.3.3 · p. 8

• 26 million medical QA pairs expand existing dataset sizes by more than two orders of magnitude.The paper describes Huatuo-26M as the largest Chinese medical QA dataset to date.Source: §6 · p. 8
• The experiments show that Huatuo-26M contains rich medical knowledge helpful to existing datasets and tasks.The authors hope it will promote medical QA research and practically help doctors and patients.Source: §6 · pp. 8–9

• The 26-million-pair scale makes expert manual checking nearly impossible, so the dataset might contain wrong medical information.The authors propose online correction and regular updates by clinical doctors or experts.Source: Limitation · p. 9
• Translation into other languages might introduce additional errors, and traditional Chinese medicine differs from Western medicine.Source: Limitation · p. 9
• Medical treatment suggestions depend on individual factors and may require multi-turn dialogue rather than single-turn QA.The cited factors include age, gender, accompanying symptoms, and symptom timing.Source: Limitation · p. 9

• Potential inaccuracies arise from doctors' mistakes in online consultations and automatic extraction of QA pairs.Patients may provide incomplete information, while the dataset is too large for comprehensive expert checking.Source: Ethics Statement · p. 9

• Encyclopedias and knowledge bases provide full-text question-answering pairs, while online consultation records offer URL and full-text versions.Source: Dataset Download · p. 9

A Word Clouds for Huatuo-26M Dataset

• Online consultation records use more informal daily language, whereas other sources contain more professional medical vocabulary, making the dataset linguistically diverse.Source: A Word Clouds for Huatuo-26M Dataset · p. 11

B Examples of Huatuo-26M Dataset

• Consultation questions are colloquial with targeted answers, while wiki and expert-article questions are concise with detailed professional answers and knowledge-base questions are concise and accurate.Source: B Examples of Huatuo-26M Dataset · p. 11

C Extracting QA pairs from encyclopedia page

• Wikipedia pages are parsed for structured information, then titles linked to paragraphs are converted into answerable questions using manually designed templates.Source: C Extracting QA pairs from… · p. 11

D Knowledge Bases Questions Templates

• The knowledge-base templates encode entity relations or attributes across 43 question types, and manual inspection found nearly every sampled QA pair correct.Source: D Knowledge Bases Questions Templates · p. 11

E Examples of Retrieval Based Benchmark

• DPR was selected because it had the best overall retrieval performance among the evaluated models.Source: E Examples of Retrieval Based Benchmark · p. 11
• Retrieved answers were generally relevant and valid, although redundant consultation records could produce false negatives and relatively low recall 5.The authors describe retrieval quality as moderately satisfied, especially for the first consultation-record case.Source: E Examples of Retrieval Based Benchmark · p. 11
• The benchmark presents DPR retrieval examples for encyclopedia questions, knowledge-base questions, and consultation-record questions.Source: Table 11 · p. 16, Table 12 · p. 16, Table 13 · p. 17