Source-linked AI summary
CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark
Ningyu Zhang, Mosha Chen, Zhen Bi, Xiaozhuan Liang, Lei Li, Xin Shang, Kangping Yin, Chuanqi Tan, Jian Xu, Fei Huang, Luo Si, Yuan Ni, Guotong Xie, Zhifang Sui, Baobao Chang, Hui Zong, Zheng Yuan, Linfeng Li, Jun Yan, Hongying Zan, Kunli Zhang, Buzhou Tang, Qingcai Chen
TL;DR
Most biomedical language-understanding benchmarks are limited to English, leaving Chinese biomedical text without a dedicated evaluation resource. The paper introduces CBLUE from real-world biomedical data across diverse tasks and evaluates 11 Chinese pre-trained models. The results show that current models remain far below human performance, while Chinese linguistic characteristics make the tasks challenging.
Problem
Most biomedical language-understanding benchmarks are English-based, despite Chinese biomedical text having distinct linguistic characteristics and lacking a dedicated benchmark.
Method
The paper constructs the first comprehensive CBLUE benchmark from real-world biomedical data, covering eight tasks, and evaluates 11 Chinese pre-trained language models.
Results
Current models perform far below human performance on CBLUE, and analyses identify challenges associated with Chinese biomedical language understanding.
Takeaways & Limitations
CBLUE provides an open, community-driven platform for Chinese BioNLP evaluation and leaves clear room for future model improvements.
Takeaways & Limitations
Chinese biomedical language is colloquial and polysemous, creating task difficulty that requires more robust models.
Abstract
from arXiv · showhide
Artificial Intelligence (AI), along with the recent progress in biomedical language understanding, is gradually changing medical practice. With the development of biomedical language understanding benchmarks, AI applications are widely used in the medical field. However, most benchmarks are limited to English, which makes it challenging to replicate many of the successes in English for other languages. To facilitate research in this direction, we collect real-world biomedical data and present the first Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark: a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification, and an associated online platform for model evaluation, comparison, and analysis. To establish evaluation on these tasks, we report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling. Our benchmark is released at \url{https://tianchi.aliyun.com/dataset/dataDetail?dataId=95414&lang=en-us}.
1 Introduction
CBLUE addresses the lack of Chinese biomedical language-understanding benchmarks by introducing a comprehensive, diverse-task evaluation resource and reporting model evaluations that remain below human performance.
- Motivation: Most biomedical language-understanding benchmarks are English-focused, while Chinese has distinct linguistic characteristics and previously lacked a dedicated benchmark.This gap matters because Chinese speakers represent a substantial global population and English-centric evaluation does not directly cover Chinese biomedical language understanding.
- Contribution: CBLUE is introduced as the first comprehensive Chinese biomedical language-understanding benchmark with eight tasks.The tasks span named entity recognition, information extraction, clinical diagnosis normalization, short-text classification, question answering, intent classification, and semantic similarity.
- Findings: Current models perform far below single-human performance, leaving substantial room for future improvement.The authors also analyze cases to identify challenges and linguistic differences in Chinese biomedical language understanding.
- Contribution: The benchmark is designed as an open-ended, community-driven platform that encourages new dataset contributions for the Chinese BioNLP community.The work also aims to support a universal GLUE-like open platform for evaluation and future research.
- Evaluation: The study systematically evaluates 11 Chinese pre-trained language models and releases baseline source code as a research toolkit.The toolkit is intended for future research purposes.
2 Related Work
Prior work established broad biomedical and general-language benchmarks, but most biomedical resources remained English-based; CBLUE fills this gap with a Chinese-specific benchmark tailored to Chinese biomedical text.
- General benchmarks: General-language benchmarks such as GLUE, SuperGLUE, DecaNLP, and SentEval enabled standardized comparison across language-understanding tasks.Non-English examples include RussianSuperGLUE and the community-driven Chinese general-domain benchmark CLUE.
- Biomedical benchmarks: Biomedical resources including PubMedQA, BioRead, and BLURB support tasks such as question answering, machine reading comprehension, and biomedical language understanding.Biomedical corpus annotation requires expert intervention and is labor-intensive and time-consuming.
- Research gap: Most existing biomedical benchmarks are based on English, so other languages may contain valuable information relevant to generalization that these resources do not capture.The related work frames multilingual coverage as an unresolved benchmark gap.
- CBLUE: CBLUE focuses on Chinese because Chinese biomedical text differs linguistically from English and has domain characteristics requiring a benchmark designed specifically for Chinese.The study therefore aims to develop the first Chinese biomedical language-understanding benchmark.
3 CBLUE Overview
CBLUE is an eight-task Chinese biomedical language understanding benchmark designed around diverse tasks, data sources, distributions, and evaluation settings. Its construction combines real-world data collection with expert annotation and quality control.
- Diverse tasks: CBLUE contains eight biomedical language understanding tasks spanning token-level, sequence-level, and sequence-pair settings.
- Data sources: CBLUE draws from clinical trials, electronic health records, medical forums, textbooks, online questions, and search-engine logs, while preserving real-world distributions.
- Quality control: Datasets are annotated or reviewed by domain experts, with majority voting, control questions, worker review, and interrater agreement checks.
- Task coverage: The benchmark covers entity recognition, relation extraction, diagnosis normalization, clinical-trial classification, sentence similarity, intent classification, query-title relevance, and query-query relevance.
- Evaluation settings: The CHIP-STS task evaluates transfer across disease types because its testing distribution differs from its training distribution.
4 Experiments
Experiments evaluate Chinese pre-trained models across CBLUE and analyze benchmark errors and representative cases. Results show that model performance remains limited by task complexity, biomedical knowledge, and Chinese-language characteristics.
- Benchmark Results: 11 Chinese pre-trained language models are evaluated on CBLUE tasks using task-specific output layers and fine-tuning.The baselines are implemented with PyTorch and include general, whole-word-masked, n-gram-enhanced, and medical pre-trained models.
- Benchmark Results: Larger pre-trained models generally perform better, but whole word masking does not consistently improve results on CTC, QIC, QTR, and QQR.These observations indicate that several benchmark tasks require more sophisticated modeling approaches.
- Benchmark Results: ALBERT-tiny achieves performance comparable to base models on CDN, STS, QTR, and QQR, while PCL-MedBERT underperforms expectations despite its medical pretraining.The authors relate the smaller-model result to differences between pretraining corpora and Chinese medical text, and use the medical-model result as evidence of benchmark difficulty.
- Human Evaluation: Human evaluation uses trained amateur annotators and specialist gold labels, while expert annotation quality is monitored through control questions and interrater agreement.Five of six annotations show almost perfect agreement, with Fleiss’ Kappa κ = 0.9.
- Error Analysis: CMeEE errors mainly involve entity overlap, ambiguity, domain knowledge, and annotation errors; KUAKE-QIC errors frequently involve multiple triggers and colloquialism.The analysis divides CMeEE errors into six categories and KUAKE-QIC errors into seven categories.
- Case Studies: Medical pretraining helps in examples containing biomedical terminology or long sentences: PCL-MedBERT identifies entities and intent labels missed by general models.Examples include lymphocyte and neutrophil ratios, anti-toxin antibodies, and deletion, translocation, and inversion terms.
5 Conclusion
The paper presents CBLUE, evaluates 11 Chinese language representation models, and finds that current models remain far from human performance on challenging Chinese biomedical language understanding tasks.
- Conclusion: CBLUE is a Chinese biomedical language understanding benchmark used to evaluate 11 current language representation models.The benchmark is positioned as a Chinese counterpart to English benchmarks such as GLUE, SuperGLUE, and BLURB.
- Conclusion: State-of-the-art models have limited ability on challenging CBLUE tasks and remain far from human performance.This contrasts with English benchmarks where model performance is reported to match human performance.
Ethical Considerations
The benchmark data were collected with authorization, anonymized, and reviewed by provider IRB committees, but real-world sampling may introduce popularity bias and annotation errors.
- Data governance: Collected data were authorized, anonymized, and reviewed by each data provider’s IRB committee to preserve privacy.The collection agreements and privacy procedures are described alongside the benchmark release process.
- Data governance: Real-world data distributions may contain popularity bias that cannot be ignored.
- Data quality: Domain experts and doctors annotated the corpus, but some labels remain incorrectly annotated.This creates a data-quality concern for interpreting benchmark-based model selection.
- Data quality: Incorrect benchmark labels could cause real-world harm if models are selected solely by benchmark scores.
C Limitations
CBLUE broadens Chinese biomedical evaluation but remains incomplete and static, leaving gaps in task coverage and robustness to real-world challenges.
- Scope: The benchmark does not cover medical dialogue generation or medical diagnosis.
- Robustness: CBLUE is static, so strong task performance may coexist with failures on simple challenge examples and real-world scenarios.
- Availability: Some shared-task datasets become unavailable after the task ends, complicating data acquisition and future research.
- Scope: CBLUE combines CHIP shared tasks with Alibaba QUAKE search-engine data across named entity recognition, information extraction, diagnosis normalization, and classification.
E.1 Chinese Medical Named Entity Recognition Dataset (CMeEE)
CMeEE is a Chinese biomedical named entity recognition dataset that extracts nine medical entity categories from text, using expert-guided annotation and strict Micro-F1 evaluation.
- Task definition: CMeEE extracts medical entities and classifies them into nine categories, including diseases, symptoms, drugs, procedures, and departments.
- Annotation: Annotation involved 32 annotators, including medical experts, biomedical informatics experts, physicians, and computer science students.
- Annotation: The annotation process used training, trial rounds, iterative guideline refinement, independent labeling, and expert-supported disagreement resolution.
- Evaluation and format: Evaluation uses strict Micro-F1, and the data were converted to unified JSON format with provider permission.
- Dataset statistics: The dataset contains 15,000 training, 5,000 validation, and 3,000 test examples.
- Dataset statistics: Its corpus includes 938 files and 47,194 sentences spanning diseases, body parts, symptoms, and medical procedures.
E.2 Chinese Medical Information Extraction Dataset (CMeIE)
CMeIE extracts subject-predicate-object relations under a 53-schema framework, while CHIP-CDN normalizes Chinese clinical diagnoses to ICD-10 standard phrases.
- CMeIE task: CMeIE identifies entities and relations in sentences and extracts all schema-constrained subject-predicate-object triples.
- CMeIE task: The CMeIE schema contains 53 relations, comprising 10 genus relations and 43 other sub-relations.
- CMeIE annotation: Chinese segmentation validation was added to CMeIE annotation, and final inter-annotator agreement reached 0.83.
- CMeIE statistics: CMeIE data include 14,339 training, 3,585 validation, and 4,482 test examples, with nearly 75,000 triples.
- CHIP-CDN task: CHIP-CDN maps Chinese electronic medical-record final diagnoses to standard phrases from the ICD-10 Beijing Clinical Edition v601 vocabulary.
- CHIP-CDN statistics: CHIP-CDN provides 8,000 training instances and 10,000 testing instances, with the training data split into 6,000 training and 2,000 validation examples.
E.4 Clinical Trial Criterion Dataset (CHIP-CTC)
CHIP-CTC frames clinical-trial eligibility prediction as a Chinese biomedical classification task and defines a 44-category semantic classification task. Its corpus was annotated and quality-checked by biomedical annotators, with Macro-F1 used for evaluation.
- Task background: Clinical-trial eligibility prediction classifies whether a subject meets a clinical trial from eligibility criteria.The task targets automation of manual comparison between medical records and trial screening criteria.
- Task description: The CHIP-CTC task defines 44 semantic categories and predicts the correct category for each input text.
- Annotation: Three annotators labeled the corpus, including an experienced biomedical researcher and two biomedical-informatics Ph.D. candidates.
- Annotation: 20,000 criteria sentences achieved a Cohen’s kappa inter-annotator agreement score of 0.9920 before the remaining 18,341 sentences were assigned for annotation.
- Evaluation: Evaluation uses Macro-F1, based on class-specific precision and recall.
- Dataset statistics: The dataset contains 22,962 training instances, 7,682 validation instances, and 10,000 test instances.
F Experiments Details
The experiments use PyTorch and report task-specific hyper-parameters, with separate configurations for CMeEE-ER and CMeEE-RE.
- Experiments use PyTorch, and running hyper-parameters are reported in the accompanying tables.
- CMeIE experiments are separated into entity recognition and relation classification stages with distinct hyper-parameter details.
G Error Analysis for Other Tasks
The error analysis categorizes recurring model failures across several biomedical tasks, including ambiguity, domain knowledge, syntactic complexity, and annotation errors.
- Error analysis examines sampled cases from CMeIE, CHIP-CDN, CHIP-CTC, CHIP-STS, KUAKE-QTR, and KUAKE-QQR.
- Ambiguity refers to similar contexts carrying different meanings that mislead predictions.
- Biomedical terminology, complex syntax, overlapping entities, long inputs, and rare words are identified as distinct sources of prediction difficulty.
- Annotation error is defined as an incorrectly assigned label, distinguishing labeling problems from model prediction failures.
- Colloquial language and irrelevant descriptions can also mislead models, especially in search-query inputs.
- The benchmark standardizes task data and training configurations through unified JSON conversion and task-specific hyper-parameter tables.