Source-linked AI summary
CMB: A Comprehensive Medical Benchmark in Chinese
Xidong Wang, Guiming Hardy Chen, Dingjie Song, Zhiyi Zhang, Zhihong Chen, Qingying Xiao, Feng Jiang, Jianquan Li, Xiang Wan, Benyou Wang, Haizhou Li
TL;DR
Medical LLM evaluation needs objective, scalable measures that remain faithful to China’s linguistic, cultural, and medical context. CMB addresses this with exam and clinical subsets, and benchmarking found strong performance from GPT-4 and several Chinese open-source models while identifying scope limitations.
Problem
Medical LLM evaluation is challenging because specialized medical questions require professional judgment, while translated benchmarks may overlook China’s regional medical practices.
Method
CMB combines qualification-exam multiple-choice questions with expert-curated clinical consultations in a taxonomy spanning Chinese medical professions and examination stages.
Results
More than 60% accuracy was achieved by GPT-4, Qwen-72B-Chat, and Yi-34B-Chat, exceeding the cited licensing threshold.
Takeaways & Limitations
CMB provides a localized and comprehensive tool for tracking medical LLM progress in China’s medical landscape.
Takeaways & Limitations
The benchmark omits many subjects requiring multimodal capabilities, and standardized CMB-Clin evaluation may not suit diverse medical conditions.
Abstract
from arXiv · showhide
Large Language Models (LLMs) provide a possibility to make a great breakthrough in medicine. The establishment of a standardized medical benchmark becomes a fundamental cornerstone to measure progression. However, medical environments in different regions have their local characteristics, e.g., the ubiquity and significance of traditional Chinese medicine within China. Therefore, merely translating English-based medical evaluation may result in \textit{contextual incongruities} to a local region. To solve the issue, we propose a localized medical benchmark called CMB, a Comprehensive Medical Benchmark in Chinese, designed and rooted entirely within the native Chinese linguistic and cultural framework. While traditional Chinese medicine is integral to this evaluation, it does not constitute its entirety. Using this benchmark, we have evaluated several prominent large-scale LLMs, including ChatGPT, GPT-4, dedicated Chinese LLMs, and LLMs specialized in the medical domain. We hope this benchmark provide first-hand experience in existing LLMs for medicine and also facilitate the widespread adoption and enhancement of medical LLMs within China. Our data and code are publicly available at https://github.com/FreedomIntelligence/CMB.
1 Introduction
CMB addresses the difficulty of evaluating medical LLMs objectively and at scale while preserving China’s linguistic, cultural, and medical context. It combines a localized benchmark design with findings across general and medical models.
- Evaluation motivation: Medical LLM evaluation is difficult because professional manual judging is expensive and real-world hospital feedback raises ethical concerns.Subjective evaluation is especially challenging for specialized questions such as radiology.
- Localization motivation: Localized benchmarks are necessary because unified or translated standards may overlook regional medical practices, including Traditional Chinese Medicine.The paper highlights TCM concepts, diagnostic methods, and treatment resources that require cross-cultural understanding.
- Benchmark design: CMB combines CMB-Exam qualification-test questions with CMB-Clin expert-curated clinical consultations across six major categories and 28 subcategories.The dataset contains 280,839 multiple-choice questions and 74 complex clinical consultations.
- Benchmark findings: More than 60% accuracy was achieved by GPT-4 and recent open-source models such as Qwen-72B-Chat and Yi-34B-Chat, exceeding the cited licensing threshold.The benchmark also reports disparities across professional levels and knowledge areas, including TCM and Western medicine.
- Benchmark findings: Automatic evaluation using GPT-4 highly agreed with expert evaluation results.The paper presents this agreement as an observation from benchmarking multiple LLMs.
2 The Philosophy of CMB
CMB is designed around medical professionals’ assessment needs, combining objective qualification-exam questions with practical, multi-round clinical reasoning. Its coverage extends beyond pharmacy to multiple professions and medical learning stages.
- Evaluation philosophy: The benchmark selects multiple-choice questions and iterative questioning over complex medical records to assess knowledge and practical problem-solving skills.Both tasks have standard answers, providing stable performance indicators.
- CMB-Exam: CMB-Exam uses qualification exams because they provide expert-supported ground truths and a 60% accuracy benchmark tied to specific professional expertise.Its coverage includes physicians, nurses, medical technicians, and pharmacists across career stages.
- CMB-Exam: The benchmark broadens medical assessment beyond pharmacy to nursing, medical technology, and other health-related occupations.The paper argues that a pharmacist examination alone has limited occupational and knowledge coverage.
- CMB-Clin: CMB-Clin tests practical knowledge application through complex clinical diagnostic problems requiring synthesis of medical knowledge and case analysis.It is presented as the first multi-round question-answering dataset based on real, complex medical records.
- Combined framework: CMB-Exam and CMB-Clin together cover theoretical knowledge and practical skills for evaluating medical professionals’ development and medical LLM learning.The two subsets are intended to provide a comprehensive evaluation framework.
3 Dataset Creation
The dataset is constructed through a structured medical taxonomy, expert-supported source selection, and preprocessing. CMB-Clin converts complex cases into multi-turn diagnostic dialogues evaluated across multiple dimensions.
- CMB-Exam taxonomy: The CMB-Exam taxonomy covers four professions plus undergraduate and graduate examinations, organized into six categories and 174 subject categories.The structure follows medical disciplinary and examination systems.
- Data preparation: CMB-Exam data comes from publicly available examination questions and coursework with expert-provided solutions, followed by de-duplication, cleansing, OCR correction, and manual quality checks.Problematic or incorrect items identified through comments were removed.
- CMB-Exam data split: 280,839 multiple-choice questions were collected, with 400 questions per subcategory for testing and 10 per subcategory for development.Three medical specialists generated explanations for development questions to support chain-of-thought experiments.
- CMB-Clin preparation: CMB-Clin cases were screened to exclude image-dependent problems, retain sufficient diagnostic information, and remove similar diseases.The source cases came from official medical textbooks.
- CMB-Clin formulation: Each CMB-Clin case is transformed into multi-turn examiner-candidate dialogue to assess diagnostic and therapeutic capabilities.The case structure includes patient descriptions, questions, and reference answers.
- CMB-Clin evaluation: CMB-Clin evaluates responses for fluency, relevance, completeness, and proficiency after concatenating the case description with prior question-answer pairs and the current question.Questions span the consultation process, with treatment-principles questions comprising 30.09%.
4 Experiments on CMB-Exam
CMB-Exam benchmarks general and medical LLMs across professional categories, answer-only and CoT settings, and perceived difficulty levels. Results show strong Chinese general-model performance, uneven specialty and career-stage accuracy, limited prompting gains, and persistently low traditional Chinese medicine accuracy.
- General LLMs: Qwen-72B, Yi-34B, and Yi-6B exceeded GPT-4 accuracy, while most evaluated models exceeded ChatGPT on Chinese medical knowledge.The authors note that Yi-6B’s performance is notable given its smaller parameter count, while cautioning about possible data leakage.
- Medical LLMs: HuatuoGPT-II surpassed all commercial models and ranked first among medical LLMs, while DISC-MedLLM ranked second among open-source medical models after CMB-Exam-Train training.The authors attribute DISC-MedLLM’s performance improvement to deterministic medical knowledge in the training questions.
- Different Categories: Pharmacist-related questions tended to score lower than nursing-related questions, reflecting variation across clinical specialties.The paper relates this difference to intricate drug-name and indication distinctions versus the more straightforward foundational knowledge required in nursing.
- Prompt Strategies: Few-shot and CoT prompting had little effect for most domestic general and medical LLMs, although few-shot prompting improved several Chinese-support-limited and smaller models.The reported beneficiaries include Deepseek-llm, Mistral-7B, Mixtral-8x7B, Qwen-1.8B, and Yi-6B.
- Prompt Strategies: CoT prompting negatively affected low-accuracy models such as Mistral and ChatGLM-Med, potentially introducing irrelevant context into non-reasoning questions.This effect is reported specifically for CMB-Exam problems that do not require reasoning.
- Different Categories: Traditional Chinese medicine questions consistently had relatively low accuracy, indicating that additional traditional Chinese medicine data are needed.The finding is presented as part of the category and difficulty analysis of CMB-Exam.
5 Experiments on CMB-Clin
CMB-Clin evaluates medical dialogue on complex records using structured prompts and expert-aligned automatic assessment. Results reveal differences across model families, metrics, and evaluation methods.
- Prompt construction: CMB-Clin prompts combine a description, optional conversation history, and the question.Conversation history is incorporated by prepending appropriate roles to each question and reference.
- Expert and Automatic Evaluation: Three medically knowledgeable annotators scored 320 responses from 11 models across Fluency, Relevance, Completeness, and Medical Proficiency on a 1–5 scale.The responses were randomly selected across models from different tiers and evaluated with reference solutions.
- General LLMs: Yi-34B matched GPT-4, Qwen-72B ranked below GPT-4 but above ChatGPT, and other evaluated models ranked below ChatGPT on medical dialogue.Except for Deepseekllm-67B and Mixtral-8x7B, the other general LLMs showed strong dialogue capabilities on complex medical records.
- General LLMs: Domestic general LLMs performed strongly on CMB-Exam but lagged behind OpenAI models on CMB-Clin, which is closer to real scenarios.The authors recommend using both CMB-Exam and CMB-Clin during model iteration to assess capabilities.
- Medical LLMs: Medical LLMs had lower overall dialogue ability than general LLMs, while BianQue-2 performed poorly on CMB-Exam but well on CMB-Clin.The results also indicate that HuatuoGPT-II-34B and DISC-MedLLM had weaker CMB-Clin performance than their CMB-Exam results suggested.
- Different Metrics: Completeness had the lowest average value among the differentiated medical dialogue metrics, while most general LLMs exceeded 4.9 in Fluency.Many medical LLMs scored below 4.5 in Fluency, indicating weaker basic dialogue capabilities.
- Expert and Automatic Evaluation: GPT-4 automatic rankings agreed strongly with expert rankings, reaching a Spearman correlation of 0.93.The rankings differed only in a flip between GPT-4 and ChatGPT, supporting high alignment between automatic and expert evaluation.
6 Conclusion
The conclusion presents CMB as a localized and comprehensive benchmark for evaluating medical LLMs in China. It is intended to track progress and support broader use of medical LLMs in the Chinese medical landscape.
- Conclusion: CMB is tailored to China’s local cultural environment to provide a more contextualized and comprehensive medical evaluation benchmark.The benchmark addresses the local setting rather than functioning only as a competitive leaderboard.
- Conclusion: CMB serves as a tool for tracking LLM progress in medical domains.The conclusion frames this tracking function as distinct from a competitive leaderboard.
- Conclusion: The benchmark may support broader and more effective utilization of LLMs in China’s medical landscape.
Ethical Statement
The ethical statement addresses data anonymity, data quality, and concerns about exaggerated LLM abilities. It describes public, processed data and additional experiments intended to support stable evaluation.
- Data Anonymity: The study data primarily come from publicly accessible examination questions and coursework exercises processed by experts, without personal information.
- Data Quality: Data quality is supported through authoritative source screening and processing intended to reduce OCR character errors and transcription errors.
- Exaggerating the Abilities of the LLM: Multiple experiments were conducted to assess result stability and address concerns about exaggerating LLM abilities.The authors describe the experimental results as providing a possible option for evaluating open-ended questions.
Limitations
The study identifies two limitations: incomplete coverage of multimodal medical subjects and standardized CMB-Clin evaluation criteria that may not fit all real-world conditions.
- Limitations: The benchmark does not cover all medical subjects requiring multimodal capabilities for real-world issues.
- Limitations: CMB-Clin uses a standardized model evaluation method despite diverse medical conditions in real-world settings.
- Limitations: The authors advocate disease-specific criteria for assessing model performance.
A.1 Medical Benchmark
Medical benchmarks cover objective multiple-choice tasks and subjective open-ended generation, but few datasets combine both. CMB expands this combined evaluation with a larger, broader Chinese medical dataset sourced from licensing exams and authoritative textbooks.
- Objective medical benchmarks typically use multiple-choice questions, while subjective benchmarks assess open-ended replies or explanations.
- Few existing datasets encapsulate both objective and subjective medical task types.
- CMB exceeds comparable datasets in size and draws questions from the Chinese National Medical Licensing Examination and authoritative medical textbooks.
- Unlike narrower datasets focused on pharmacists or physicians, CMB includes questions spanning multiple medical careers.
- CMB is presented as a larger and more comprehensive resource than previously available datasets.
B Dataset
CMB-Exam is organized across clinical professions, career stages, examination subjects, departments, and courses, and the evaluation covers medical, general-domain, and proprietary LLMs. The dataset also includes a TCM subcatalog whose expressions differ substantially from English medicine.
- Dataset organization: CMB-Exam is structured into six primary sections by clinical professions and their commonly undertaken exams, then subdivided by career progression, subjects, departments, and courses.
- Traditional Chinese medicine: Keyword comparisons show that expressions in the CMB TCM subcatalog and English medicine are very different.
- Evaluated models: The evaluation includes 12 Chinese medical LLMs, 5 proprietary LLMs, and 13 publicly available general-domain LLMs.
- Evaluated models: The evaluated medical models include HuatuoGPT-II, BianQue, ChatMed-Consult, MedicalGPT, ChatGLM-Med, Bentsao, and DISC-MedLLM.
- Training setting: DISC-MedLLM-13B was fine-tuned on CMB-Exam-Train for evaluation.
- Evaluated models: The general-domain models include ChatGLM-3, Baichuan2-chat, Qwen-Chat, Yi, Deepseek-llm-67B-Chat, Internlm-Chat, Mistral, and Mixtral.
D Experiment Details of CMB-Clin
CMB-Clin evaluates model answers with an expert-informed automatic prompt that rates fluency, relevance, completeness, and proficiency in medicine. The experiment samples 320 responses for human evaluation.
- Automatic evaluation: The automatic-evaluation prompt specifies task instructions, metrics, criteria, and answer placeholders, and is used by both ChatGPT and GPT-4.
- Evaluation criteria: The evaluation rates fluency, relevance, completeness, and proficiency in medicine on five-point scales.
- Automatic evaluation: The prompt supplies a description, conversation, question, solution, and model answer for evaluation.
- Human evaluation: 320 model responses were randomly sampled for the experiment because the answer volume was large and expert resources were limited.
- Human evaluation: Human evaluation results are presented in Table 11.
E.2 Robustness Experiments on CMB-Clin
The CMB-Clin evaluation is robust across repeated GPT-4 judging and shows strong agreement with expert ratings. Ranking stability is assessed across decoding temperatures, with one noted exception among general models.
- Repeated evaluation: Three independent GPT-4 evaluation runs were conducted for each general and medical LLM between January 20 and January 26, 2024.
- Repeated evaluation: Medical-model rankings remained unchanged across repeated runs despite deviations in absolute scores.
- Repeated evaluation: General-model rankings also remained unchanged across repeated runs except for Yi-34B.
- Expert agreement: Pearson correlations between expert and GPT-4 evaluations exceeded 0.71 for fluency, relevance, completeness, and proficiency in medicine.
- Temperature robustness: Evaluation rankings under different decoding temperatures are compared using pairwise Spearman correlations.